How to get all items from large lists in Sharepoint Online

As you probably know Sharepoint lists have default throttling limit 5000 items which means that if query result to more than 5000 items it will throw resource throttling exception. One solution which you may try is to use RowLimit:

<View>
    <Query>
        <Where>
            <Eq>
                <FieldRef Name=\"Title\" />
                <Value Type=\"Text\">test</Value>
            </Eq>
        </Where>
    </Query>
    <RowLimit>5000</RowLimit>
</View>

Or with Camlex:

var camlQuery = Camlex.Query()
    .Where(x => (string)x["Title"] == "test")
    .Take(5000)
    .ToCamlQuery();

However if list contains more than 5000 items this query will anyway result in resource throttling exception regardless of RowLimit being set. Another possible solution is to try to add index to set some field as indexed and use condition against this field in query. But what if you can't do that (e.g. if you don't have control over the list schema)? How to get all items from large list then?

For that you may use query ID > 0 (which is always true since ID is auto generated integer) and CamlQuery.ListItemCollectionPosition property which allows to get paginated results. Code will look like this:

ListItemCollectionPosition position = null;
var items = new List<ListItem>();
do
{
    var camlQuery = Camlex.Query()
        .Where(x => (int)x["ID"] > 0)
        .ViewFields(x => new []
        {
            x["ID"],
            x["Title"]
        })
        .Take(5000)
        .ToCamlQuery();
    camlQuery.ListItemCollectionPosition = position;
    var listItems = list.GetItems(camlQuery);
    ctx.Load(listItems);
    ctx.ExecuteQueryRetry();

    items.AddRange(listItems);
    position = listItems.ListItemCollectionPosition;

    if (position == null)
    {
        break;
    }
} while (true);

It will fetch items by 5000 items per page until all items from large list will be fetched.