Skip to content

Instantly share code, notes, and snippets.

@reke592

reke592/web-notes.txt

Last active June 11, 2023 09:36
Show Gist options
  • Save reke592/c070d746a917f203d0ead4b3f91ebabf to your computer and use it in GitHub Desktop.
Save reke592/c070d746a917f203d0ead4b3f91ebabf to your computer and use it in GitHub Desktop.
Download ZIP
web notes
Raw
web-notes.txt
-- Disclaimer: not a standard rule. just a collection of ideas/experience while doing a fullstack development task --
identify the configurations that can affect the application restart, put those dynamic configuration in database. (eg. email config).
create server.js (exports app and start function), and index.js (imports start function), so we can use the server.app for unit testing with supertest.
identify the common functions and store them inside `commons` folder.
create commons/errors to contain all the possible application errors we can throw, include the http status code.
[frontend]
eg. /commons
'-- debouncer.dart
'-- errors.dart
[backend]
eg. /api/commons
'-- event_bus.js
'-- errors.js
identify all common UI components/functionalities, create a wrapper for them, put them in a separate application package, the idea is to lessen the impact of third-party library syntax update in whole application source code.
eg. commons_package
'-- /widget
| '-- data_grid.dart
| '-- form_page.dart
'-- /functions
'-- show_dialog.dart
'-- show_toast.dart
identify the application features and create a directory for each, put them inside `features` folder.
[frontend]
eg. /features
'-- /auth
'-- /applicant_masterlist
[backend]
eg. /api/features
'-- /auth
'-- /applicant_masterlist
identify all the components needed to fulfill the requirements of certain feature put them inside `features/<the_feature>` folder
[frontend] - requires refactorization
eg. /features/auth
'-- data
| '-- auth_api.dart
| '-- AuthUserModel.dart
'-- presentation
| '-- /components
| | '-- login_button.dart
| | '-- logout_button.dart
| '-- /dialogs
| | '-- session_expired_dialog.dart
| | '-- session_idle_dialog.dart
| '-- /pages
| '-- login_page.dart
| '-- signup_page.dart
'-- auth_provider.dart
[backend]
eg. /api
'-- /controllers
| '-- /handlers
| | '-- onUserCreated.js
| | '-- onUserVerified.js
| '-- auth_controller.js
'-- /models
| '-- user_model.js
'-- /services
'-- auth_service.js
[below pattern may add complexity in backend]
while designing the backend folder structure, we normally endup with 3-4 files inside a domain feature folder.
a model for data source queries,
a result set mapper to control the domain model structure
a controller to handle the http request, response and event bus subscribers
and a domain use case file that contains all the validation for domain business logic
eg. /api/features
'-- /applicants
| '-- /work_experiences
| | '-- work_experience_mapper.js
| | '-- work_experiences_controller.js
| | '-- work_experiences_model.js
| '-- /education_attainments
| | '-- education_attainment_mapper.js
| | '-- education_attainments_controller.js
| | '-- education_attainments_model.js
| '-- /basic_info
| | '-- basic_info_controller.js
| | '-- basic_info_mapper.js
| | '-- basic_info_model.js
| '-- applicants_domain_use_case.js
'-- /user
| '-- user_mapper.js
| '-- users_controller.js
| '-- users_model.js
| '-- users_domain_use_case.js
'-- /user-types
| '-- user_type_mapper.js
| '-- user_types_controller.js
| '-- user_types_model.js
...
Much better to keep the structure simple. Utilize the IDE / code editor fetatures to locate the files that we need.
/backend
'-- /database
| '-- /deployments
| | '-- /v1
| '-- /schema
'-- /logs
| '-- /rotation
| | '-- 2023-11-06.rar
| | '-- 2023-11-05.rar
| '-- out.log
| '-- err.log
'-- /src
'-- /config
| '-- default.json
| '-- prod.json
| '-- test.json
'-- /customs
| '-- /client1_customs
| '-- /client2_customs
'-- /startup
| '-- config_migrations.js
| '-- customizations.js
| '-- cron_jobs.js
| '-- routes.js
'-- /api
| '-- controllers
| | '-- /handlers
| | | '-- onUserCreated.js
| | | '-- onUserVerified.js
| | '-- login_controller.js
| | '-- signup_controller.js
| '-- middlewares
| | '-- auth.js
| '-- models
| | '-- user_model.js
| '-- services
| '-- login_service.js
| '-- signup_service.js
'-- /test
| '-- test.spec.js
'-- server.js
'-- index.js
Don't worry about the structure of response data, flat JSON struction from SQL query results are fine, use mapper to create nested JSON object response only when needed.
(eg. third party integrations, expose different endpoint for them, avoid re-using the standard backend controllers. treat them as client customizations)
Avoid querying database with default parameters in DAL just to avoid parameter errors, it is good to identify the script lapses than to get blinded by fail-safe process.
eg.
```
// will not throw undefined id error
// hence the procedure validation returns all when id is null
undefToNull = (value) => value == undefined ? null : value;
return await DB.query('CALL sp_get_json_replies(:form_id, :question_id, :created_by_user_id, :current_user_id);', {
replacements: {
form_id,
question_id: undefToNull(question_id),
created_by_user_id: undefToNull(created_by_user_id),
current_user_id,
},
transaction
});
// better approach
strNull = (value) => `${value}` == 'null' ? null : value;
return await DB.query('CALL sp_get_json_replies(:form_id, :question_id, :created_by_user_id, :current_user_id);', {
replacements: {
form_id,
question_id: strNull(question_id),
created_by_user_id: strNull(created_by_user_id),
current_user_id,
},
transaction
});
```
Handling dynamic model in SQL storage
[schema]
- identify the aggregation, (what the models have in common)
- use separate table per subclass, connect them using static value like sub_type_id
eg. tbl_form_content_types // definitions of subclass
tbl_form_contents
'-- tbl_instructions // sub_type_id 1
'-- tbl_sections // 2
'-- tbl_questions // 3
[saving]
- save/update per subclass (to maximize the control for subclass parameter requirements)
- create a post script to update the commons table (aggregate root)
[querying the final result]
- use temporary table
- insert each sub class into temp table, convert the results directly to json (eg. JSON_OBJECT in MySql, FOR JSON in MSSql)
We need to create a separate package for the auth related functions in order to modularize the whole project and customizations.
this practice will result to a per-scope development, wherein we can create different features and less thinking about the main project source code.
Handling client custom packages (part 1) [eg. flutter project]
- as we mention, we are talking about packages, meaning we need to create a separate package per client to contain all the customizations.
- for this we need 2 packages, `projX_commons` and `projX_customs`
- the client package must require the project commons, this common package contains all the interface exposed by the main project. also works like a bridge to connects the main project to other packages.
package:projX_commons
'-- interface/
| '-- IProjectXHttp
| | '-- get
| | '-- post
| | '-- put
| | '-- delete
| '-- IProjectXSession
| | '-- getCurrentSession
| | '..
| '-- IProjectXModuleRegistry
| '-- register(ModuleMetaInf)
| '..
'-- models/
'-- SessionTokenModel (readonly JSON claims)
| - avoid binding json claims to class property, dependent packages does'nt need to know everything
'-- ModuleMetaInf (module_name, module_icon, module_path, bottom_nav_entry, side_nav_entry, etc.. main proj layout entry)
- package:projX_customs
- requires service container package (eg. in flutter we can use Get_it service locator)
- requires package:projX_commons
- package:projX
- requires package:projX_commons
- requires package:projX_customs
- create a concrete class for IProjectXHttp, commonly this class will inject the session token to each HTTP request and also manage the backend address. put the singleton instace to our service container.
- create a concrete class for IProjectXSession, we can initialize the instance when there is auth session or its up to the requirements, again we need to feed this to our service container.
- create a concrete class implementation of IProjectXModuleRegistry and again feed the class instance to our service container
- make the projX install the modules inside the concrete implementation of IProjectXModuleRegistry, its like spreading the module meta information to dedicated locations in our projX.
- finally, if there is client custom, we need to re-create the package projX_customs (MUST have thesame package name)
- we will use the service locator to access the object instance created by the main projX (eg. GetIt.instance<IProjectXHttp>())
- and then we override the main project dependency projX_customs
- as a result, we separate the client custom source codes to the main project source codes.
Handling client custom packages (part 2) [eg. nodejs backend]
- the main idea is to create a contract / finite format, like each custom index.js entry must have the below properties in module exports
* name : String - the client custom package name
* registerRoutes({app}) : Function
- inside this function we use the app to register the main routes
* registerMiddlewares({app, login_middlewares:[], logout_middlewares:[]}) : Function
- inside this function we can add middlewares to the app (eg. to intercept controller function and inject the custom validations)
- also we have the login_middlewares:[] wherein we can add the third-party hooks on user login, this is to provide the third-party session token to client.
- finally the logout_middlewares, to destroy the previously created third-party session.
- in javascript source codes, we will use the `require.main.require` to require the main project related stuff (eg. node_modules installed in main project, api models, commons, etc..)
- just in case we need to use other node_modules, we can initialize a new package in the client custom folder, so we can install the requirements. for this we will use the normal `require` method to refer in local directory node_modules folder.
as much as possible choose a Ubiquitous language that anyone can understood, ask the domain experts (the client), seek suggestions from developers, testers, qa. finally everyone must agree.
eg. given we are working on a domain for Job posting
JobsProvider.getList() <-- not clear
JobRequisitionProvider.getList() <-- much better
avoid returning plain text in api response, much better to use a JSON object. create a finite model for plain message response.
eg. { name: 'error', code: 'ERCODE_XXXX' message: 'foo' }
centralize the route definitions in one file, so we can see the possible conflicts when using route params.
when using external libraries always create a wrapper function, so we can inject our custom feature.
always propagate the error by throwing, centralize the catch so we can easily control the application output.
run heavy task in background (separate process), configure (optional) a socket endpoint to monitor the status.
never include any business logic rules in controller, always have a use-case builder function, it is more readable with DI pattern support.
avoid populating the controller, implement a message bus, emit the event after business logic call to all subscribers.
folder by feature is great for low-coupling high cohesion source codes, but we also need to consider how big is the project.
folder by feature is great for UI development.
when using flutter, flutter-modular is a great library combined with providers for state handling.
frontend development requires separation of concerns eg. VIEW - PROVIDER - MODELS - API
store access token in memory, use cookie for refresh token, on page-reload revalidate using the refresh token
on specs gathering always ask the client how they want to receive the results, is it immediate (realtime) or eventually (on specific schedule).
when dealing with repositories, we need an object mapper to freeze the results, also to have a finite domain model schema.
when overriding response value, we need to put them in controller.
we might get confused sometimes while designing a feature,
eg. job_post, applicant, job_application.
- job_post and applicant can be considered as plain model
- job_application on the other hand is more likely a service, because it has the business logic that connects the applicant and job_post
- much better if we have separate controller and repository per each model and service.
when designing data encryption, each aggregate entity MUST NOT share the same encryption key.
eg. store the private key in database
- we can also encrypt the private key in database, as a result there is no way to decrypt the data using the plain database record.
- each user must have a unique key IV for encryption
- store the decrypted value in result cache
- just incase the user want to delete all the private details related to his account, we simply remove the key IV and cache entry.
never use a websocket library alone, always use a redis store so it can run in cluster / load balancers.
avoid relying too much in SQL ORM libraries, we can use them for normal select and update, but for complex queries, use stored procedure.
use Postman for API documentation and testing, most of the time we can spot the security issues by testing in plain request.
in backend, eval is evil.
apache,nginx,express,php etc... anything in server configuration, always hide the version / server details.
use pm2 to monitor nodejs service, it can also run node process in cluster, auto-reload the service when it crash or reached the memory limit.
always configure log-rotation, archive them properly.
search for updated list of security headers.
when using reverse proxy setup in nginx/apache, we need to upgrade the connection for websocket endpoints, in normal http it close the connection after transmision, having it not upgraded the client will use http everytime we emit on socket.
when assigning server hostname, we can use the hosts file to test the name resolution in local eg. 127.0.0.1 sample.domain.com.
use separate file for each host configuration, store them in directory ./sites-enabled.
always test the new configuration before restarting any service. unless you want some adrenaline rush. jk. ALWAYS test before restart.
system batch file / shell script is your friend.
on team discussion, always consider the posibilities that someone might not yet familiar in the tech-stack that we use.
google is your friend, avoid mastering everything, they always update the technology, look for the documentation/manual.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment