Best approach for web applications

Mark81

Hello,

I've already developed several web-based application using Qt5 as engine for data processing. The webpages are accessed from remote terminal, often from smartphones. Here my current approach:

on the server machine (often a small SBC) I run any http server, i.e. lighttpd
I develop the user interface in html5 and javascript - to handle actions and behavior of the UI
on the server runs also a Qt5 application which provides the communications with the outside world (database, local I/O, fieldbus, etc...)
both the webpages and the "core" Qt5 application exchange data using a websocket. In this way I can transmit and receive data between the user side and server side

This approach works but has several drawbacks:

it's not so easy (for me at least) to provide server side code to make a really dynamic pages. I found two ways: 1. send the generated html code through the websocket and use a javascript function to put it into the actual page, 2. use some server side language (i.e. cgi) and create another communication channel between it and the Qt5 application. Currently I create the contents directly from javascript, but this is not so safe
the websocket(s) requires at least to open one more port than the default 80 and that should be forwarded to the http server
for the same reason it's hard to have more servers in the same LAN because you have to forward both ports (http and websocket ones) to the correct local ip

That said, I have few questions:

in general, this is the best approach or you might suggest a better one?
any hint about the drawbacks?
until now the webservers were designed to answer to only one client, but in the next project the web-application should be used by dozen of clients at the same time (the goal is to acquire some data and put it into a db, taking care of which user submitted). I guess it's much better to put the webpages on a reliable third-party hosting provider, but then I'm afraid I also need to set up my own server to run the Qt5 application

Thanks for any suggestion!

kshegunov

@Mark81
Hi,
I always found REST to be the easiest to implement server side with a language such as C++. It's possible to have, for example HTML + dojo store that's backed by a real simple HTTP server in Qt. You won't need the websockets this way, only handling the few basic HTTP requests: put, get, delete. And since this returns JSON objects as data (which is also handled transparently by dojo), you don't really need to inject them into the JS code. This is how you can do it with Qt, however as far as I can tell the newest and hottest in web dev is node.js, so perhaps you should research that.

Kind regards.

tekojo

Not really my area at all, but the two I have looked at a bit are Cutelyst http://cutelyst.org/
and Silk http://silk.qtquick.me/

Both are on Github if you want to take a look. https://github.com/cutelyst/cutelyst and https://github.com/qt-users-jp/silk

Mark81

Thanks both for inputs.
All those tools seem great but I need to ask if you would rely on third-party code for production.
I mean, Qt Framework is actively maintained and has a huge community behind. Is it the same for dojo/cutelyst/silk? I don't know, hence the question :-)

It's quite hard to decide which tool is the best for my applications. Do you recommend any website or even plain book to better understand the differences among them without spending days to study/try each one?

tham

Did you give wt a try?

kshegunov

The thing is web dev changes every 2 years or so, so whatever you choose it probably will be old in a couple of years. I would rely on established codebases for production (known and actively maintained JS libs/frameworks), nothing fancy. For example, depending on your requirements and bear in mind it's not a complete list: jquery, dojo, prototype. They have different learning curves and applicability. I personally think that dojo is best thought out, but most complex to get into as well.

Never used any of the other stuff (cutelyst, silk, wt) to give an even near to competent opinion.

I hope that helps somewhat.
Kind regards.

Mark81

@kshegunov I agree with you approach. All that technologies are fancy but I cannot guess their reliability and long-term availablility/stability. I'm going to better understand what is dojo, I've already used jquery and prototype by the way.

Mark81

@kshegunov I'm reading further about dojo on its website. Do you know any simple (but complete) "hello world" example that involves also Qt5? I mean to understand how all the characters work together in order to create a web application?

kshegunov

To be hones I haven't stitched together dojo with Qt, only dojo with some PHP code and it was a few years back. However, the thing about dojo (probably the other JS frameworks nowadays too) is that it's oblivious of the backend. It makes use of the http protocol only - from the backend site you get GET, POST, DELETE with appropriate headers and respond in JSON. As I said, while I think it possible, I challenge the idea of using Qt for a web app, not because Qt isn't suited it just might require more effort than with other specialized technologies.

Do you know any simple (but complete) "hello world" example that involves also Qt5? I mean to understand how all the characters work together in order to create a web application?

No I don't have anything on hand, but I'd start with taking one of the dojo examples (e.g. a table) and redirecting the backend to a Qt based server. That'd be the easiest thing in my opinion.

ssokolow

The first thing you should be thinking about is architecture.

You need some scalable way to service HTTP connections.
From the server's perspective, the whole point of using WebSockets instead of raw TCP is that it can share port 80 or port 443 with HTTP requests to work around overzealous corporate firewalls... as long as the HTTP server supports it.

You should be able to use the same solution that's standard for deploying popular web frameworks like Rails, Django, etc.:

Run a web server like nginx (pronounced "engine X")
Configure it to handle TLS termination if you want TLS (You don't want to risk misconfiguring your TLS. Doing TLS properly with nginx is well-documented.)
Serve any non-CDN [1] [2] [3] static assets directly from it
Configure it to reverse-proxy (eg. /api -> http://localhost:8080/ ) connections for dynamic content to one or more application servers (eg. your Qt WebSocket server, uWSGI, etc.) sitting behind your firewall.

Here's their documentation on WebSocket proxying.

You'll still need your Qt code to be scalable, though. The general rules for that when writing an application server are:

Don't store any state. Let the developers of your database/memcached/etc. handle the problem of ensuring consistent data at scale so you can just run multiple copies of your code in parallel when demand exceeds what a single core or server can handle.
Use a load-testing tool like ab (HTTP), JMeter (HTTP, WebSocket, etc.), or thor (WebSocket) to stress-test your server and, if the results aren't good enough, use a profiler (like OProfile, perf, or gprof on Linux) to figure out where it's bottlenecking before you start changing things.
If you're writing your application server from scratch rather than using a ready-made "call this callback whenever a request/message is received" application server that handles the vagarities of parallel/distributed execution for you, know the ups and downs of different architectures.

I tend to use the select() approach when I need to write a scalable server from scratch... especially outside Windows where select() isn't limited to TCP/UDP sockets. If you want to use the thread-per-client approach, see this StackOverflow answer.

Mark81

@kshegunov said in Best approach for web applications:

No I don't have anything on hand, but I'd start with taking one of the dojo examples (e.g. a table) and redirecting the backend to a Qt based server. That'd be the easiest thing in my opinion.

Well, I agree it should be the easiest thing but I will be bound to my own little HTTP server... I guess it would be better if I can rely on any other third-part server. In this way in the future I can also move the pages to a remote hosting provider, while leaving the logic backend elsewhere.

Mark81

@ssokolow said in Best approach for web applications:

From the server's perspective, the whole point of using WebSocket instead of raw TCP is that it can share port 80 or port 443 with HTTP to work around overzealous corporate firewalls... as long as the HTTP server supports it.

Thank you very much for your analisys. I need to read further about some stuff you said. But now I would ask you a clarification about this point.
The bottom line of all the previous posts is: I can avoid the use of (web)sockets only if I develop my own HTTP server as kshegunov suggests. Otherwise the only way to exchange data between the client and the "backend" is the (web)socket, isn't it?

Perhaps I still have some confusion about the architecture. Please, correct me if I'm wrong!
In my mind a web-based application is composed of 3 parts:

client, usually a web browser that shows the html pages and executes the client-side code (i.e. javascript)
HTTP server, which answers to the browser requests
backend, the "logic" application (written in any language, of course!) which is the engine of the web application. It is the piece of code that communicates with the "outside world", i.e. using serials, local I/O, databases etc...

For small applications the HTTP server and the backend software might run on the same machine (i.e. the SBC of the first post). But for larger projects the web pages could be hosted on a remote server, while the backend application may run on the office server, near the machines or other stuff.

Websockets allow the client to communicate directly with the backend, regardless the physical position of the HTTP server.
Instead, if you embed the HTTP server in your backend application you can avoid the websocket because you directly handle the client HTTP requests.

ssokolow

@Mark81 WebSockets aren't about letting the client talk to another server.

You can always use CORS (Cross-Origin Resource Sharing) to allow a browser to talk to multiple machines at once.

You can even use the PeerConnection feature of WebRTC to allow your server to matchmake direct browser-to-browser connections capable of transferring arbitrary data.

WebSockets are for allowing the client and server to hold a connection open so the server can send push events to the client, rather than having the client poll the server for updates. (That's why a chat client or other real-time communication application is the most popular type of WebSockets demo)

Whether the client can talk directly to the backend is purely a characteristic of how you design your system.

kshegunov

@Mark81 said in Best approach for web applications:

t I will be bound to my own little HTTP server...

Well from your original posting I assumed this is what you want.

I guess it would be better if I can rely on any other third-part server

Which is what I hinted in my first post; no need to reinvent the wheel. If I wanted some Qt based processing, I'd rise a "server frontend" (e.g. apache + PHP/Perl/CGI/Ruby or whatever) in one of the web-specialized languages and communicate over TCP or w/e with a Qt server that provides the service I want. This also allows to move the webservice frontend around hosting companies ...

Mark81

@kshegunov ok now I understand! Thanks!