Paging results gives you an extra page

[pdehaan]

I was testing on stage and I had 6 total records and noticed that when I paging results (?_limit=2), and get results 5 and 6, the readinglist server returns a header link for Next-Page, which returns 0 results.

---

Page 1 of 3 (limit == 2):

$ http GET https://readinglist.stage.mozaws.net/v1/articles?_limit=2 --auth="pdehaan:"

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified
Backoff: None
Connection: keep-alive
Content-Length: 799
Content-Type: application/json; charset=UTF-8
Date: Thu, 09 Apr 2015 21:44:26 GMT
Last-Modified: 1428613604938
Next-Page: https://readinglist.stage.mozaws.net/v1/articles?_limit=2&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzNTk0MzkwfQ%3D%3D
Total-Records: 6

{
    "items": [
        {
            "added_by": "a",
            "added_on": 1428613604939,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "5e35f9ff-b15d-43bf-ba6e-55e1313af54c",
            "is_article": true,
            "last_modified": 1428613604938,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "git://h",
            "stored_on": 1428613604939,
            "title": "d",
            "unread": true,
            "url": "git://h",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613594390,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "6d5bbcae-5d49-4f29-bd42-0523cbf9530d",
            "is_article": true,
            "last_modified": 1428613594390,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "ftp://h",
            "stored_on": 1428613594390,
            "title": "d",
            "unread": true,
            "url": "ftp://h",
            "word_count": null
        }
    ]
}

---

Page 2 of 3 (limit == 2):

$ http GET https://readinglist.stage.mozaws.net/v1/articles?_limit=2\&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzNTk0MzkwfQ%3D%3D --auth="pdehaan:"

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified
Backoff: None
Connection: keep-alive
Content-Length: 805
Content-Type: application/json; charset=UTF-8
Date: Thu, 09 Apr 2015 21:46:40 GMT
Last-Modified: 1428613604938
Next-Page: https://readinglist.stage.mozaws.net/v1/articles?_limit=2&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzNTY5Njc3fQ%3D%3D
Total-Records: 6

{
    "items": [
        {
            "added_by": "a",
            "added_on": 1428613582376,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "acd04f35-2855-4e8f-8241-46a8c6bf8978",
            "is_article": true,
            "last_modified": 1428613582375,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "https://h",
            "stored_on": 1428613582376,
            "title": "d",
            "unread": true,
            "url": "https://h",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613569677,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "69a276ef-8551-4bc4-961a-360062794108",
            "is_article": true,
            "last_modified": 1428613569677,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "http://h",
            "stored_on": 1428613569677,
            "title": "d",
            "unread": true,
            "url": "http://h",
            "word_count": null
        }
    ]
}

---

Page 3 of 3 (limit == 2):

$ http GET https://readinglist.stage.mozaws.net/v1/articles?_limit=2\&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzNTY5Njc3fQ%3D%3D --auth="pdehaan:"

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified
Backoff: None
Connection: keep-alive
Content-Length: 847
Content-Type: application/json; charset=UTF-8
Date: Thu, 09 Apr 2015 21:47:43 GMT
Last-Modified: 1428613604938
Next-Page: https://readinglist.stage.mozaws.net/v1/articles?_limit=2&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzMjc3ODMyfQ%3D%3D
Total-Records: 6

{
    "items": [
        {
            "added_by": "a",
            "added_on": 1428613495359,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "15a575d6-c4d3-4197-811a-a765bdc393b6",
            "is_article": true,
            "last_modified": 1428613495358,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "https://locljost/2",
            "stored_on": 1428613495359,
            "title": "d",
            "unread": true,
            "url": "https://locljost/2",
            "word_count": null
        },
        {
            "added_by": "pdehaar",
            "added_on": 1428613277831,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "8c567b34-1ebd-4034-9963-ebea95bef1e6",
            "is_article": true,
            "last_modified": 1428613277832,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "s",
            "resolved_url": "https://locljost",
            "stored_on": 1428613277831,
            "title": "s",
            "unread": true,
            "url": "https://locljost",
            "word_count": null
        }
    ]
}

---

Page 4 of 3 (limit == 2)

$ http GET https://readinglist.stage.mozaws.net/v1/articles?_limit=2\&_token=eyJsYXN0X21vZGlmaWVkIjoxNDI4NjEzMjc3ODMyfQ%3D%3D --auth="pdehaan:"

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified
Backoff: None
Connection: keep-alive
Content-Length: 12
Content-Type: application/json; charset=UTF-8
Date: Thu, 09 Apr 2015 21:49:16 GMT
Last-Modified: 1428613604938
Total-Records: 0

{
    "items": []
}

Actually, it'd odd that Total-Records returned here is 0 instead of 6.

[pdehaan]

ALL RESULTS:

$ http GET https://readinglist.stage.mozaws.net/v1/articles --auth="pdehaan:"

HTTP/1.1 200 OK
Access-Control-Expose-Headers: Backoff, Retry-After, Alert, Next-Page, Total-Records, Last-Modified
Backoff: None
Connection: keep-alive
Content-Length: 2429
Content-Type: application/json; charset=UTF-8
Date: Thu, 09 Apr 2015 21:43:48 GMT
Last-Modified: 1428613604938
Total-Records: 6

{
    "items": [
        {
            "added_by": "a",
            "added_on": 1428613604939,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "5e35f9ff-b15d-43bf-ba6e-55e1313af54c",
            "is_article": true,
            "last_modified": 1428613604938,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "git://h",
            "stored_on": 1428613604939,
            "title": "d",
            "unread": true,
            "url": "git://h",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613495359,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "15a575d6-c4d3-4197-811a-a765bdc393b6",
            "is_article": true,
            "last_modified": 1428613495358,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "https://locljost/2",
            "stored_on": 1428613495359,
            "title": "d",
            "unread": true,
            "url": "https://locljost/2",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613582376,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "acd04f35-2855-4e8f-8241-46a8c6bf8978",
            "is_article": true,
            "last_modified": 1428613582375,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "https://h",
            "stored_on": 1428613582376,
            "title": "d",
            "unread": true,
            "url": "https://h",
            "word_count": null
        },
        {
            "added_by": "pdehaar",
            "added_on": 1428613277831,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "8c567b34-1ebd-4034-9963-ebea95bef1e6",
            "is_article": true,
            "last_modified": 1428613277832,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "s",
            "resolved_url": "https://locljost",
            "stored_on": 1428613277831,
            "title": "s",
            "unread": true,
            "url": "https://locljost",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613569677,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "69a276ef-8551-4bc4-961a-360062794108",
            "is_article": true,
            "last_modified": 1428613569677,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "http://h",
            "stored_on": 1428613569677,
            "title": "d",
            "unread": true,
            "url": "http://h",
            "word_count": null
        },
        {
            "added_by": "a",
            "added_on": 1428613594390,
            "archived": false,
            "excerpt": "",
            "favorite": false,
            "id": "6d5bbcae-5d49-4f29-bd42-0523cbf9530d",
            "is_article": true,
            "last_modified": 1428613594390,
            "marked_read_by": null,
            "marked_read_on": null,
            "preview": null,
            "read_position": 0,
            "resolved_title": "d",
            "resolved_url": "ftp://h",
            "stored_on": 1428613594390,
            "title": "d",
            "unread": true,
            "url": "ftp://h",
            "word_count": null
        }
    ]
}

[Natim]

It is then intended behavior when the last page is full we don't have any efficient ways to know it is the last page.

Maybe we can add a counter of past records on the continuation token to fix this.

[leplatrem]

Thanks for this feedback!

Note for later: https://github.com/mozilla-services/cliquet/blob/1.6.0/cliquet/tests/resource/test_pagination.py#L98-L103