2020-02-16 Firefox syncserver

OK, so I’m trying to run my own syncserver. It’s separate from the account server, so you still need a Firefox Account. If you do things in the wrong order your browser syncs immediately – to their own sync server, which is not what I wanted. 😭

Anyway, let’s start from the beginning.

First, check out the project, syncserver. It uses Python 2. Oh well!

The Sync Server software runs using python 2.7, and the build process requires make and virtualenv.

I looked at this blog post by Matthias Dietel.

apt install python-dev git-core python-virtualenv g++

Easy enough! Build the project:

cd ~/src
git clone https://github.com/mozilla-services/syncserver.git
cd syncserver
make build
make test

Edit the syncserver.ini file. I’ve made the following changes:

diff --git a/syncserver.ini b/syncserver.ini
index ccf1ae0..21d0586 100644
--- a/syncserver.ini
+++ b/syncserver.ini
@@ -1,6 +1,6 @@
 [server:main]
 use = egg:gunicorn
-host = 0.0.0.0
+host = localhost
 port = 5000
 workers = 1
 timeout = 30
@@ -11,7 +11,7 @@ use = egg:syncserver
 [syncserver]
 # This must be edited to point to the public URL of your server,
 # i.e. the URL as seen by Firefox.
-public_url = http://localhost:5000/
+public_url = https://alexschroeder.ch/sync
 
 # By default, syncserver will accept identity assertions issued by
 # any BrowserID issuer.  The line below restricts it to accept assertions
@@ -20,7 +20,7 @@ public_url = http://localhost:5000/
 identity_provider = https://accounts.firefox.com/
 
 # This defines the database in which to store all server data.
-#sqluri = sqlite:////tmp/syncserver.db
+sqluri = sqlite:////home/alex/syncserver.db
 #sqluri = pymysql://sample_user:sample_password@127.0.0.1/syncstorage
 
 # This is a secret key used for signing authentication tokens.
@@ -34,7 +34,7 @@ identity_provider = https://accounts.firefox.com/
 
 # Set this to "false" to disable new-user signups on the server.
 # Only requests by existing accounts will be honoured.
-# allow_new_users = false
+allow_new_users = new
 
 # Set this to "true" to work around a mismatch between public_url and
 # the application URL as seen by python, which can happen in certain reverse-
@@ -55,4 +55,4 @@ force_wsgi_environ = false
 # MySQL based syncstorage-rs 1.5 server hosted at http://localhost:8000/1.5
 
 # "{node}/1.5/{uid}"
-# sync-1.5 = "http://localhost:8000/1.5/{uid}"
+sync-1.5 = "https://alexschroeder.ch/sync/token/{node}/sync/1.5/{uid}"

That is to say:

  1. the server only listens on localhost:5000
  2. the server can be reached at https://alexschroeder.ch/sync
  3. it uses a SQLite db at /home/alex/syncserver.db
  4. it allows new users to sync their stuff
  5. and I had to change that endpoint at the end... 🤷

So now I need to set up my website. I use Apache and I already have proxy stuff set up. All I had to add was this:

ProxyPass /sync	    http://localhost:5000/

Thus, any request starting with /sync to my existing website gets sent to port 5000 on localhost which happens to be the URL where the sync server is running.

Next, open about:config in the browser and change identity.sync.tokenserver.uri to https://alexschroeder.ch/sync/token/1.0/sync/1.5 – then I’m ready to open the browser preferences and sign in.

If everything worked, you should see your display name in the top spot of the hamburger menu. 🍔 😁

If you see a warning sign after a second or two, stuff isn’t working.

It all sounds so easy now, but it took me a while to get everything working, believe me. 😭

Now I’m seeing the following errors in the log output of make serve:

ERROR:syncserver:The public_url setting doesn't match the application url.
This will almost certainly cause authentication failures!
    public_url setting is: https://alexschroeder.ch/sync
    application url is:    http://localhost:5000/sync
You can disable this check by setting the force_wsgi_environ
option in your config file, but do so at your own risk.

I have no idea what it means. I’m thinking it works anyway? I’m making an additional change to syncserver.ini just to be sure.

force_wsgi_environ = true

The instructions tell me that we only need this in certain reverse-proxy setups but this is a reverse-proxy setup so I think I should be fine?

Now I wonder how to leave the sync server running... Perhaps it’s OK to just do this every now and then?

Now that I have it running I wonder about the utility of it all:

  1. there’s no need for me to mix my private Firefox and my work Firefox bookmarks and logins
  2. I currently cannot make Firefox on my iPad and my iPhone use a different sync address

Oh Apple, the golden prison, where I’m safe, but also where I’m locked in. Sigh. 😔

But wait… This issue is closed: Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync #5006?

One of the last comments on that issue, by user fireglow:

Just to chime in a little: Python 2.7, the flavor of Python the sync software is written in, will go End-of-Life at the end of this month, year, and decade. Mozilla already has indicated that there will be no rewrite for Python 3. I gather there’s a rewrite of these services in Rust in the works, at https://github.com/mozilla-services/syncstorage-rs It’s as of now unclear to me how all these parts will fit together in a way so us self-hosters will be able migrate over.

Oh wow.

I think I’m going to stop all of this. It’s making my head hurt. I checked the Rust rewrite project. Apparently you need to use either MySQL or Spanner as your DB. And with that, I’ve decided that not running this service and not syncing my stuff is probably for the best.

I undid all the changes, stopped the server, uninstalled the software, and deleted the Firefox account again.

Tags:

Comments

Hi Alex,

You can set a custom sync server amd token server in Firefox ios.

To access it:

  • Enter the settings menu
  • Hit the version number 5 times, this opens debug mode
  • Under the sync option, there is a new menu item, Advanced Sync
  • The two settings are now available

Note that I haven’t done this, as I don’t (yet) have a sync server.

dgold 2020-02-16 19:25 UTC


I get the feeling that this doesn’t work on the iPad... I get a bunch of extra options but no place where I can set the sync server URL.

– Alex Schroeder 2020-02-16 19:28 UTC


I don’t know what to say to you. I set up my server in the way you described (except I didn’t set the last two options in the file)

I accessed the Advanced Settings in FFox on iPadOS. All works perfectly.

dgold 2020-02-16 20:24 UTC


How weird. I just updated Firefox on the iPad and it does look different, but I still don’t know where to start. Do you feel like pasting a screenshot, or can you explain it like I’m super confused? Because I feel like I am!

  1. open Firefox on the iPad
  2. click hamburger menu
  3. scroll down until you see “Version 22.0 (17157)
  4. tap this section five time until the list of options changes
  5. now what?

At the top, I see three items:

  1. my email address and a note saying I need to enter my password (but since I deleted my account, clicking on this item just takes me to accounts.firefox.com where I could probably recreated my account and start sync immediately but with their servers)
  2. error diagnosis: ask for an upgrade (translating from German)
  3. error diagnosis: forgot sync-status (translating from German)

No other item in these settings seem related to sync. What am I not seeing?

– Alex Schroeder 2020-02-16 22:08 UTC


After you tap the section until the lost of options changes, there should be a new menu item at the top of the Preferences under “Firefox Account” called “Advanced Sync Settings”.

In that modal dialog you can set the address of your sync server, in the form https://example.com/sync/token/1.0/sync/1.5

I didn’t create a FFox account until after I’d created my sync server.

dgold 2020-02-17 11:00 UTC


Oh wow, I just tried it on the iPhone and it worked. Tapped on the Version five times, it automatically scrolls to the top showing me the things I listed previously, but now I scroll back to the bottom and do it again and now I see new items at the top:

  1. Sign in to Sync
  2. Advanced Sync Settings
  3. Debug: use stage servers

Hah, thanks for pointing me at it again. I might give this another try.

– Alex Schroeder 2020-02-17 11:40 UTC


Well, I think it’s working:

Firefox on iOS Advanced Sync Settings Firefox on iOS Advanced Settings

On the iPad, the situation was different. I had to do the following:

  1. log into sync even though I couldn’t change the sync server and Firefox synced with the default sync server
  2. log out of sync
  3. activate the advanced settings again
  4. now you can change the sync server
  5. log into sync again

🙄

I’m looking at Run your own Sync-1.5 Server right now and I’m wondering about the WSGI setup. Are you using it?

I guess I don’t need to use it as my setup is basically Apache + Gunicorn. 😀

Except have to turn on force_wsgi_environ. As far as I can see in the Apache documentation on Reverse Proxy Request Headers but then again, the nginx setup instructions mention X-Forwarded-Proto and X-Real-IP, both of which the Apache docs don’t mention. Gaaaaah! 😭

Setup for Monit:

check process syncserver matching syncserver
    start program = "/bin/bash -c '(cd /home/alex/src/syncserver && exec /usr/bin/make serve 2&>1 syncserver.log)' &"
    stop program = "/usr/bin/killall gunicorn"
    if failed
	host alexschroeder.ch
	port 443
	type tcpssl
	protocol http
	request "/sync"
	for 5 cycles
	then restart
    if totalmem > 100 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then stop

– Alex Schroeder 2020-02-17 21:19 UTC


thank you very much for your post! i learned a lot here!

one question: does your setup still work on your iPad?

i have the following sync (token) server running: https://github.com/crazy-max/docker-firefox-syncserver

but i cant get past the following issue:

https://github.com/mozilla-mobile/firefox-ios/issues/6535

all the best, ma

– ma 2020-07-31 10:05 UTC


I don’t remember seeing any error messages about OAuth. In fact, I don’t remember seeing any errors at all. It was all silent fails.

– Alex 2020-07-31 13:03 UTC


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Just say HELLO