Fitter - Open Source no-code tool for map-reduce data from different source and even more!

Hello everyone!

I am Pxyup and today i want represent for you by open source project Fitter.

PxyUp / fitter

New way for collect information from the API's/Websites

Fitter Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

1. Server - parsing response from some API's or http request(usage of http.Client)
2. Browser - emulate real browser using chromium docker playwright/cypress and get DOM information
3. Static - parsing static string as data

Format which can be parsed

1. JSON - parsing JSON to get specific information
2. XML - parsing xml tree to get specific information
3. HTML - parsing dom tree to get specific information
4. XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter

go get github.com/PxyUp/fitter

…

View on GitHub

How it was created

In 2023, I worked on an idea called Trip Searcher:

go get github.com/PxyUp/fitter

The Trip Searcher would monitor flights and return potential routes from the starting city, including total prices to various destinations, and send notifications to Telegram with:

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

For this setup, I needed a list of countries, cities, and airport codes to plug into the sites mentioned. During development, I started thinking about how convenient it would be if all this information could be easily combined and parsed to streamline requests, which led to the idea for a project I call Fitter.

P.S.: This project was for personal use.

Fitter CLI

A no-code map-reducer that returns data in user-friendly (JSON) or custom formats, suitable for storage in a database or transmission via HTTP.

Features:

1. Supports parsing through HTML (query), JSON (gjson), XML, and xpath parsers.
2. Retrieves data as a browser would, using Docker, Playwright( stealth mode), HTTP Client, Cache, File, or propagated fields, with support for custom plugins.
3. Provides proxy support for Playwright and HTTP clients.
4. Can send or store information to a file, webhook, console, and more via plugins.
5. Handles all data types: int, float, bool, array, object, null, and string.
6. Combines (map-reduce) and transforms fields.
7. Utilizes the powerful expr library for template syntax, which is available across the application.
8. Offered as a standalone binary and Docker version.
9. Allows limits on request counts or instances for browser/Docker usage.

Examples

Static generation:

Here we will just generate static array from hardcoded

1.  You enter a budget.
2.  You specify a starting city or country.
3.  You set the trip duration and a range of possible start and end dates.

1.  Flight costs (parsed from Google or Kiwi).
2.  Hotel prices (from Airbnb or Booking).
3.  Food costs (retrieved from Numbeo).

Get current time

Get information from the website and return to user.

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_static_connector.json

[
        "PAGE: 1 INDEX: 0",
        "PAGE: 2 INDEX: 1",
        "PAGE: 3 INDEX: 2",
        "PAGE: 4 INDEX: 3",
        "PAGE: 5 INDEX: 4"
]

Get current Steam Sales

That will create sales.md

wile in provided directory

Get best news from HackerNews Comment list for each

In this config we using template syntax for propagate result from the first request to next one.

go get github.com/PxyUp/fitter

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

Scrape all images from website and store them locally

1.  You enter a budget.
2.  You specify a starting city or country.
3.  You set the trip duration and a range of possible start and end dates.

1.  Flight costs (parsed from Google or Kiwi).
2.  Hotel prices (from Airbnb or Booking).
3.  Food costs (retrieved from Numbeo).

Fitter

Fitter it is extended version of the Fitter CLI which have:

1. Http server for trigger
2. Return response as telegram message/webhook
3. And that currently did not have documentation :)

Usage

This tools can be used in different purpose:

1. Web scrapper
2. Data scrapper with plugins
3. Produce specific load testing
4. Build chat bots - i use it for automate my telegram channel

For example this job every day send best Dev.to aritcles:

Plans

Add more browser tools. Like click/scroll(currently can be done only by JS injection)
Improve template syntax
Add custom template editor config editor
May be will think about SASS for fitter CLI for run custom workflow for customers and return result to APP/Watch/etc.

Really wait for your feedback! Ask any question i will ask

The above is the detailed content of Fitter - Open Source no-code tool for map-reduce data from different source and even more!. For more information, please follow other related articles on the PHP Chinese website!