Web API – return huge dataset via many records

Possibly save 1 hour of your time: My Web API returns many records and it breaks after it returns some where above 2 million records.

I tried the following but it did not work. According to documentation, it only works if you are using the JavaScriptSerializer Class. The max value is based on Int32.

At this moment, I don’t have any solution yet. The only thing I can think of is that the consumer can hit the database directly if accessible rather than calling the API. Or break down the query on the API to smaller data sets and retrieve partition subsets and aggregate them again.

Of course, you have to reason whether to bring all data back to memory and store on server cache is gonna help with front end performance. This will also require huge memory footprint on the server.

If you don’t need all data at once and can do specific query to minimize the number of data records, this might also perform better rather than trying to deal with all data at once.

  [Test, MaxTime(90000)]
        public void GetLotsOfData_ViaHttpCallThroughWebApi()
        {
            string apiUrl = "https://yourapi.com/api/data";

            HttpClient client = new HttpClient();
            var response = client.GetAsync(apiUrl).ConfigureAwait(false);
            var results = response.GetAwaiter().GetResult();

            Assert.That(results.IsSuccessStatusCode);
        }
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s