Unintended Behaviour | Using a Website as your API
Pitch
I am not the target audience of TikTok, but I am still able to enjoy some of the content. Usually I share some of the found videos with my friends. I like to do it by downloading the video and sending that to them. This way they can watch it more conveniently especially if they are not using the app. However there are some videos which cannot be downloaded within the app.
When I noticed this, I did what everyone else would: opened my laptop, went to the site, looked at the HTML content, copied the video url, opened it in a new tab, then downloaded it. Now I could send it as usual. This works, but it’s not convenient, so that’s when the idea came: Could I do that on my phone with a push of a button?
The obvious choice on Android is the share / other feature because then my App can listen to the intent. I pitched my idea to some of my friends, it made sense. After that I looked into Play Store if somebody already done this. Well, of course they have. The idea is good so there are already lots of apps for it, one with over 5 million downloads released more than a year earlier than I even thought of the idea.
So that’s it, there is no reason for me to writing it. Well, at least doesn’t make sense to release it, but writing it? That can be a good exercise. In this article I would like to explain my workflow of using the website in an unintended way and what steps I usually take.
Android App
As stated above the Android side is pretty simple, so just let’s go over it quickly. First I need an intent filter to capture the share event from TikTok. Looking over which apps show up when clicking on the share action, the assumption is clear, it is text / message. With that I already have an entry point in our code and the url which directs us to the site.
Next, to make it seamless, I used a Foreground service to do the actual data loading. That way I can just continue scrolling through the videos while my App does its thing. To make sure even if an error happens I can still retry the request, I save the urls into SharedPreferences.
So it was time for me to create an actual UI where the app shows the loading and has a retry action, and that’s about it. Here comes the fun part, how can we get the actual video from that URL?
Inspection
Obviously, what I need to do is the same thing I did manually. Load the url, search for the video in the received HTML and just request the video. At least that’s what I thought. In the age of everything JavaScript, where not even my Bank’s site works without JS, it’s not that easy. The received HTML doesn’t yet contain the actual video, its loaded afterwards. Even so, first thing one could try is to disable JS and see if the website loads differently thus getting the data rather easily. Once JS is disabled I saw that the website actually does what I would expect, only a logo can be seen and nothing much really.
If not the easy way, then …
Even so, using a browser means we have access to its debugging tools to replicate the same request it does. I’m using chrome based browser, so with inspect I can go backwards from the end video to how can I get to it. Right click, inspect, network and refresh the site. Wow, now that’s a lot of calls, however going to the elements I can copy the video URL and filtering with that on the Networks tab, I have my end goal, the request data of the video.
Now we have the specific request I need to call to get the video file, with the help of the browser, we can replicate the same request just by copying its headers and parameters and adding it into a Retrofit Service. I like replicating the requests in a test file. That way I can run them directly on my device, no need to build the Android app, but it lets me experiment with the same code I will use later. So with a bit of experimentation I can replicate that request just like the browser would. This is how that looks.
You may ask what’s the returned VideoResponse and how it’s parsed. Well it’s just a simple model parsed with a custom Retrofit Converter.Factory. I used this to separate the parsing components, nothing else.
Now I have the last request I need to call, but how can we get there?
More inspection …
Going back to Chrome inspection. Looking at the request, I can select Initiator. Inspecting that we see there are no intermediary requests, it’s started just from the first request sent to the server, which is great, less hoops to jump through.
Next, still on the Network panel, even though there is a filter, pressing control F / command F, we can have a more detailed search. Putting the first part of the video request into the search. Where we can find how the url is received in our browser. With this we can see that it is indeed coming in the first request’s HTML response with some specific keyword just before it.
Testing
Since I need a couple of experimentation to ensure my parser is indeed working, what I like to do is copy a couple of responses into simple HTML files. Use MockWebServer to mimic the request response and experiment with my parsing until I am sure I get the proper URL and nothing else. I do this so I don’t do actual requests to the server, on one hand, that would be slower, on second hand, I don’t want to have unnecessary network traffic.
The tests are up and running against the copied responses, now I can verify it’s indeed working against the real server.
Tinkering…
Well, of course it doesn’t, usually these things happen because I missed a header or cookie or something similar. The best course of action is to compare what’s in the browser’s request against what I have just sent. So a little bit of tinkering experimentation and questioning, why I am even doing this, and it’s done. Now I can replicate the request series to get the video.
I wrote a test against the real server as well with a locally saved video comparing the files contents, just so I have a way to quickly verify if everything is still working. Since it’s not my backend I won’t be notified if something changes. However such test is shaky, it depends on network connection and also I wouldn’t want to run a request with every one of my unit tests, so I put Disabled annotation over it and only run it occasionally.
So with all that work, I have the means to get the URL, the means to convert that URL to the actual video file, and all that remains is to tie these two together. Save the video in a way that it can be accessed by the gallery and I am done. But the enemy comes and its name is Captcha.
Circumvent Captcha? No.
During the tinkering with the URL one inevitably will run into a Captcha. The reason is that too many requests are sent to the server. One could think that with the same principles used before the Captcha can be circumvented.
Maybe with enough tinkering we could. We can add break points into the javascript, see the initiator of the request and maybe replicate the same logic to fool the Captcha, but I would choose not to. Remember the goal is to download a couple of videos while scrolling through TikTok, I don’t want to load all of their video collection.
During normal use, this issue probably won’t happen. Even if it does, I don’t want to send more requests than the server wants us to. Since we already save the urls into SharedPreferences, we can easily retry them later when my calls are no longer too heavy for the Server.
Conclusion
Following these steps, anyone can convert a website into an App for their own usage. However, one should never abuse this, bringing down the website with hundreds and thousands of requests is not good neither for you neither for them, so be mindful about that.
For me this was a good exercise, especially since it’s the first time I finally upgraded to JUnit5 from JUnit4.
I hope it gave you a bit of insight how can you see what’s happening inside your browser and replicate it. Or at least sparked your curiosity what could you automate yourself.
Repository
If you are interested in the actual source, you can find it on GitHub, by clicking here. I tried to have better code quality than my last project, but be aware, I’m still doing this in my spare time so I’m not that focused on this. Also be aware I am no designer and have pretty bad eye for it.
Demo
Now go code. ;)
