Evidence of any pre-roman occupation in Caister is sparse. Four pieces of Irish gold found in the 1950's and a hoard of bronze items found during the construction of the by-pass in the late 1970's are not enough to indicate a settlement but they do indicate that man was present or passing through this area. They might also indicate an export route to the continent via the port at that was sited at Caister at that time.
Caister was once part of an island, later called the Isle of Flegg, and was situated on the northern edge of a large estuary. Most of this estuary has now silted up and Great Yarmouth has been built upon it, Breydon water is a small surviving part of this once vast estuary. In around AD125 a new port was founded on the northern edge of the estuary. The Romans constructed a fort covering approx. 9 acres. The fort was used to protect the estuary and was doubtless intended for developing trade with the Rhineland.
Hairpins, bracelets and rings have been found at this fort so it is evident that both women and children lived with the soldiers.
The fort was occupied for about 200 years but was left deserted in the 5th century when the Romans withdrew from Britain

Anglo Saxons then occupied Caister. By about 680 Christianity was well established in Norfolk, thanks to the arrival of St. Augustine in 587 in Kent from Rome. He converted King Aethelbert to Christianity. A new community had built up around the old Roman fort.

A vast burial has been found in Caister holding an estimated 3000-4000 bodies, it stretched from south of the Roman fort to the site of the present day church
In the 9th C there were constant raids, firstly from Mercia to the West and then, increasingly, from Denmark and Norway.
The superb seamanship and fighting qualities of the Scandinavians made them formidable enemies and also East Anglia was now the smallest and weakest English Kingdom. By the middle of the 9th century there were large-scale Viking landings.
The king of the East Anglia was captured, tortured and beheaded. From 870-920 East Anglia was under Danish rule.
The Danes formed a settlement closer to the sea shore, laying the foundations for the present day Caister.
By 917 Norfolk was once again under English control with the Danish defeated by Edward the Elder.

The east coast, especially the area of the Fleggs, stood out as distinctive region in the Domesday Book. The population was 2-3 times greater as in the west of the county, and the farming was intensive, keeping large flocks of sheep, presumably on the marshes. Abundant meadows are described, which may possibly have been in the low-lying area that later became the Broads. Salt production was also an important industry in this area and the Domesday book records 45 salt pans in Caister. The population of Caister at that time was around 500.

The Fastolf family moved into Caister in the 14th century and became major landowners. John Fastolf was born in 1378 in the manor house in the west of the village. He proved to be a great soldier, finally earning a knighthood in 1418.
Sir John obtained a license to build himself a 'fortified house' and the old manor house of his birth was demolished in 1432 to make way for the new building. The 'castle' became one of the richest and finest building of it's type in England. Sir John moved in in 1454 at the grand age of 76. On the 5th November 1459 he passed away having only lived in the castle for 5 years.
Fastolf left a complicated will. The castle was passed on to the Paston family (of Paston letters fame) who lived there until 1599. In 1659 the castle was sold to a London moneylender to pay off a Paston debt. The castle was left to ruin, although Caister Hall was occupied by various families until the mid 20th century.

The population of Caister remained fairly constant and in 1603 there were 250 adults.