Improving my semi-automated browser finance app logins

Disclaimer: this is a work in progress

GitHub Repo (dev branch currently)

I say semi-automated because the issue of 2FA and Captcha still requires human intervention. It is possible to bypass those but I don’t address it for my needs.

So a while back I worked on this project which uses Puppeteer to login to a bank account and even factors in 2FA to get the account balance and write it to a spreadsheet. I never added more than that initial BofA account.

This is an extension of that. Still uses Puppeteer under the hood but has a general structure/pattern so that multiple accounts can be added. The main thing that’s new is using a Chrome extension to assist with the steps/targeting for the scraping and inserting into a DB for an API to use. Overall this will be used as a base for my personal “finance engine” if you will. Personally knowing if I’m f’d financially is comforting/a form of sanity for me.

Current code demo

Design

This is not going to be super polished, I need to finish it in the weekend, preferably in a day (did not finish). It’s possible since most of this work is just grunt work, not really discovery.

The main things to consider are:

• shared JSON pattern of accessing/scraping a website
• 2FA handling and async/out of order processing
• repeatability, ideally I can keep hitting this multiple times a day
• a base to build from eg. Google Spreadsheet as a database
• Chrome extension to help make it easier to add more accounts

In this implementation I am not using Twilio as a bridge. I’m using a basic website since I have web-texting and my email opened on my desktop.

It’s possible you could write an Android app that detects the incoming SMS, I’ve seen some apps do that where it just pops up and asks you to copy it or automatically inserts it into the app pin prompt.

Remarkable 2

So you can see the four main parts above.

Here you can see the “account stack” where it has a bunch of accounts to login to and it’ll process through them all. I think I will make them independent of each other (write when ready) so that if any fail, I still have some written down.

Here is an example JSON structure for the account access.

You can see that there is a website URL to go to. The interactions are done in order (array). For 2FA when I receive one I add it to the auth_codes database table and then the login script polls the API for this particular auth code until it gets it. Then Puppeteer continues with the auth and account access process. Lastly you’ll see a spreadsheet_column so the entry can be written in the right cell.

Of course the processor should be able to handle failures if any of the steps fail.

The Chrome extension is basic, I’m just going for the manual route to add steps. The main reason for the Chrome extension is the interactive DOM targetting. I was working on an automated “one click record” deal here. But it needs work.

Also the interface will just be open by default unless one is in progress then it persists the data until saved/cleared. I’m using localStorage to persist state in the CE popup UI until written to db.

Combined frontend UI

App components and tech stack

This application will have four main parts:

• Chrome extension
• API
• Small frontend website for the API
• Google Spreadsheet interface

Tech stack is all JavaScript eg. Node along with Puppeteer and Google Spreadsheet API. This is hosted locally, probably going on one of my local network RPis.

Development thoughts

This was one of my first drafts for the Chrome Extension popup interface. I wanted to just hit record and it would automatically capture every event, that’s not impossible but harder to do.

It then starts to log events like this:

This could be insecure since it’s executing in the website context. With regard to typing a password

But for now I’ll use discrete events. It’s helpful for targets that have obfuscated CSS selectors, in which case hopefully using something like input[type=email] is good enough.

Here is the manual input Chrome extension.

As I work on this I realize I’m reinventing Selenium web driver.

The reason I’m using a .env for account creds is if I were to do it through Chrome Extension/save it in database, I have to think about at rest encryption/decryption during the app runtime. Which I’ve done before with PHP, for Node I have to look into that. It might be better if you used some secret manager service.

Yeah unfortunately it can use more work on the “hands off” automation part. If you run into captcha issues or the 2FA.

If you reload a chrome extension you will need to refresh any page that has an injected script from the extension. Particularly with regard to communication.

Haha weee

The funny thing thing is this took me say 2, 8 hour days to build this (and it’s not done). And so if each login takes about 10 minutes, it will take just under 100 logins before this starts to pay off in terms of time spent so far. Also it seems like on average I do the logins about 5 times a month so it would take almost a year and a half for this to be worth doing. Oh well… good practice I guess.