yakumo.izuru/yukari
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

コードのリファクタリングが完了しました

Browse Source 
Signed-off-by: Izuru Yakumo <yakumo.izuru@chaotic.ninja>

git-svn-id: https://svn.yakumo.dev/yakumo.izuru/yukari/trunk@147 f3bd38d9-da89-464d-a02a-eb04e43141b5
This commit is contained in:
yakumo.izuru 2024-03-18 01:23:54 +00:00
parent c647ddbdf1
commit af1d8662af
17 changed files with 411 additions and 329 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Makefile
View File

@ -19,7 +19,7 @@ GOOS ?= linux
all: yukari
yukari: vendor
$(GO) build $(GOFLAGS) -o $@
$(GO) build $(GOFLAGS) ./cmd/yukari
clean:
$(RM) -f yukari
install:

26
README.md
View File

@ -1,25 +1,25 @@
# Yukari
# Yukari's Gap
Web content sanitizer proxy as a service, fork of [MortyProxy](https://github.com/asciimoo/morty) with some suggestions from the issue tracker applied, named after [Yes, that Gap Youkai](https://en.touhouwiki.net/wiki/Yukari_Yakumo)
Web content sanitizer proxy as a service, fork of [MortyProxy](https://github.com/asciimoo/morty) with some suggestions from the issue tracker applied, named after [the youkai you shouldn't ever come near](https://en.touhouwiki.net/wiki/Yukari_Yakumo)
Yukari rewrites web pages to exclude malicious HTML tags and attributes. It also replaces external resource references to prevent third party information leaks.
Yukari's Gap rewrites web pages to exclude malicious HTML tags and attributes. It also replaces external resource references to prevent third party information leaks.
The main goal of yukari is to provide a result proxy for [searx](https://asciimoo.github.com/searx/), but it can be used as a standalone sanitizer service too.
The main goal of this tool is to provide a result proxy for [searx](https://asciimoo.github.com/searx/), but it can be used as a standalone sanitizer service too.
Features:
- HTML sanitization
- Rewrites HTML/CSS external references to locals
- JavaScript blocking
- No Cookies forwarded
- No Referrers
- No Caching/Etag
- Supports GET/POST forms and IFrames
- Optional HMAC URL verifier key to prevent service abuse
* HTML sanitization
* Rewrites HTML/CSS external references to locals
* JavaScript blocking
* No Cookies forwarded
* No Referrers
* No Caching/Etag
* Supports GET/POST forms and IFrames
* Optional HMAC URL verifier key to prevent service abuse
## Installation and setup
Requirement: Go version 1.10 or higher.
Requirement: Go version 1.16 or higher.
```
$ go install marisa.chaotic.ninja/yukari@latest

58
cmd/yukari/allowed_content_types.go Normal file
View File

@ -0,0 +1,58 @@
package main
import (
"marisa.chaotic.ninja/yukari/contenttype"
)
var ALLOWED_CONTENTTYPE_FILTER contenttype.Filter = contenttype.NewFilterOr([]contenttype.Filter{
// html
contenttype.NewFilterEquals("text", "html", ""),
contenttype.NewFilterEquals("application", "xhtml", "xml"),
// css
contenttype.NewFilterEquals("text", "css", ""),
// images
contenttype.NewFilterEquals("image", "gif", ""),
contenttype.NewFilterEquals("image", "png", ""),
contenttype.NewFilterEquals("image", "jpeg", ""),
contenttype.NewFilterEquals("image", "pjpeg", ""),
contenttype.NewFilterEquals("image", "webp", ""),
contenttype.NewFilterEquals("image", "tiff", ""),
contenttype.NewFilterEquals("image", "vnd.microsoft.icon", ""),
contenttype.NewFilterEquals("image", "bmp", ""),
contenttype.NewFilterEquals("image", "x-ms-bmp", ""),
contenttype.NewFilterEquals("image", "x-icon", ""),
contenttype.NewFilterEquals("image", "svg", "xml"),
// fonts
contenttype.NewFilterEquals("application", "font-otf", ""),
contenttype.NewFilterEquals("application", "font-ttf", ""),
contenttype.NewFilterEquals("application", "font-woff", ""),
contenttype.NewFilterEquals("application", "vnd.ms-fontobject", ""),
})
var ALLOWED_CONTENTTYPE_ATTACHMENT_FILTER contenttype.Filter = contenttype.NewFilterOr([]contenttype.Filter{
// texts
contenttype.NewFilterEquals("text", "csv", ""),
contenttype.NewFilterEquals("text", "tab-separated-values", ""),
contenttype.NewFilterEquals("text", "plain", ""),
// API
contenttype.NewFilterEquals("application", "json", ""),
// Documents
contenttype.NewFilterEquals("application", "x-latex", ""),
contenttype.NewFilterEquals("application", "pdf", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.text", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.spreadsheet", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.presentation", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.graphics", ""),
// Compressed archives
contenttype.NewFilterEquals("application", "zip", ""),
contenttype.NewFilterEquals("application", "gzip", ""),
contenttype.NewFilterEquals("application", "x-compressed", ""),
contenttype.NewFilterEquals("application", "x-gtar", ""),
contenttype.NewFilterEquals("application", "x-compress", ""),
// Generic binary
contenttype.NewFilterEquals("application", "octet-stream", ""),
})
var ALLOWED_CONTENTTYPE_PARAMETERS map[string]bool = map[string]bool{
"charset": true,
}

9
cmd/yukari/link_http_equiv_safe_values.go Normal file
View File

@ -0,0 +1,9 @@
package main
var LINK_HTTP_EQUIV_SAFE_VALUES [][]byte = [][]byte{
// X-UA-Compatible will be added automaticaly, so it can be skipped
[]byte("date"),
[]byte("last-modified"),
[]byte("refresh"), // URL rewrite
[]byte("content-language"),
}

23
cmd/yukari/link_rel_safe_values.go Normal file
View File

@ -0,0 +1,23 @@
package main
var LINK_REL_SAFE_VALUES [][]byte = [][]byte{
[]byte("alternate"),
[]byte("archives"),
[]byte("author"),
[]byte("copyright"),
[]byte("first"),
[]byte("help"),
[]byte("icon"),
[]byte("index"),
[]byte("last"),
[]byte("license"),
[]byte("manifest"),
[]byte("next"),
[]byte("pingback"),
[]byte("prev"),
[]byte("publisher"),
[]byte("search"),
[]byte("shortcut icon"),
[]byte("stylesheet"),
[]byte("up"),
}

297
yukari.go → cmd/yukari/main.go
View File

@ -4,6 +4,7 @@ import (
"bytes"
"crypto/hmac"
"crypto/sha256"
_ "embed"
"encoding/base64"
"encoding/hex"
"errors"
@ -27,6 +28,7 @@ import (
"golang.org/x/net/html/charset"
"golang.org/x/text/encoding"
"marisa.chaotic.ninja/yukari"
"marisa.chaotic.ninja/yukari/config"
"marisa.chaotic.ninja/yukari/contenttype"
)
@ -37,8 +39,6 @@ const (
STATE_IN_NOSCRIPT int = 2
)
const VERSION = "v0.2.1"
const MAX_REDIRECT_COUNT = 5
var CLIENT *fasthttp.Client = &fasthttp.Client{
@ -46,137 +46,6 @@ var CLIENT *fasthttp.Client = &fasthttp.Client{
ReadBufferSize: 16 * 1024, // 16K
}
var cfg *config.Config = config.DefaultConfig
var ALLOWED_CONTENTTYPE_FILTER contenttype.Filter = contenttype.NewFilterOr([]contenttype.Filter{
// html
contenttype.NewFilterEquals("text", "html", ""),
contenttype.NewFilterEquals("application", "xhtml", "xml"),
// css
contenttype.NewFilterEquals("text", "css", ""),
// images
contenttype.NewFilterEquals("image", "gif", ""),
contenttype.NewFilterEquals("image", "png", ""),
contenttype.NewFilterEquals("image", "jpeg", ""),
contenttype.NewFilterEquals("image", "pjpeg", ""),
contenttype.NewFilterEquals("image", "webp", ""),
contenttype.NewFilterEquals("image", "tiff", ""),
contenttype.NewFilterEquals("image", "vnd.microsoft.icon", ""),
contenttype.NewFilterEquals("image", "bmp", ""),
contenttype.NewFilterEquals("image", "x-ms-bmp", ""),
contenttype.NewFilterEquals("image", "x-icon", ""),
contenttype.NewFilterEquals("image", "svg", "xml"),
// fonts
contenttype.NewFilterEquals("application", "font-otf", ""),
contenttype.NewFilterEquals("application", "font-ttf", ""),
contenttype.NewFilterEquals("application", "font-woff", ""),
contenttype.NewFilterEquals("application", "vnd.ms-fontobject", ""),
})
var ALLOWED_CONTENTTYPE_ATTACHMENT_FILTER contenttype.Filter = contenttype.NewFilterOr([]contenttype.Filter{
// texts
contenttype.NewFilterEquals("text", "csv", ""),
contenttype.NewFilterEquals("text", "tab-separated-values", ""),
contenttype.NewFilterEquals("text", "plain", ""),
// API
contenttype.NewFilterEquals("application", "json", ""),
// Documents
contenttype.NewFilterEquals("application", "x-latex", ""),
contenttype.NewFilterEquals("application", "pdf", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.text", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.spreadsheet", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.presentation", ""),
contenttype.NewFilterEquals("application", "vnd.oasis.opendocument.graphics", ""),
// Compressed archives
contenttype.NewFilterEquals("application", "zip", ""),
contenttype.NewFilterEquals("application", "gzip", ""),
contenttype.NewFilterEquals("application", "x-compressed", ""),
contenttype.NewFilterEquals("application", "x-gtar", ""),
contenttype.NewFilterEquals("application", "x-compress", ""),
// Generic binary
contenttype.NewFilterEquals("application", "octet-stream", ""),
})
var ALLOWED_CONTENTTYPE_PARAMETERS map[string]bool = map[string]bool{
"charset": true,
}
var UNSAFE_ELEMENTS [][]byte = [][]byte{
[]byte("applet"),
[]byte("canvas"),
[]byte("embed"),
[]byte("math"),
[]byte("script"),
[]byte("svg"),
}
var SAFE_ATTRIBUTES [][]byte = [][]byte{
[]byte("abbr"),
[]byte("accesskey"),
[]byte("align"),
[]byte("alt"),
[]byte("as"),
[]byte("autocomplete"),
[]byte("charset"),
[]byte("checked"),
[]byte("class"),
[]byte("content"),
[]byte("contenteditable"),
[]byte("contextmenu"),
[]byte("dir"),
[]byte("for"),
[]byte("height"),
[]byte("hidden"),
[]byte("hreflang"),
[]byte("id"),
[]byte("lang"),
[]byte("media"),
[]byte("method"),
[]byte("name"),
[]byte("nowrap"),
[]byte("placeholder"),
[]byte("property"),
[]byte("rel"),
[]byte("spellcheck"),
[]byte("tabindex"),
[]byte("target"),
[]byte("title"),
[]byte("translate"),
[]byte("type"),
[]byte("value"),
[]byte("width"),
}
var LINK_REL_SAFE_VALUES [][]byte = [][]byte{
[]byte("alternate"),
[]byte("archives"),
[]byte("author"),
[]byte("copyright"),
[]byte("first"),
[]byte("help"),
[]byte("icon"),
[]byte("index"),
[]byte("last"),
[]byte("license"),
[]byte("manifest"),
[]byte("next"),
[]byte("pingback"),
[]byte("prev"),
[]byte("publisher"),
[]byte("search"),
[]byte("shortcut icon"),
[]byte("stylesheet"),
[]byte("up"),
}
var LINK_HTTP_EQUIV_SAFE_VALUES [][]byte = [][]byte{
// X-UA-Compatible will be added automaticaly, so it can be skipped
[]byte("date"),
[]byte("last-modified"),
[]byte("refresh"), // URL rewrite
[]byte("content-language"),
}
var CSS_URL_REGEXP *regexp.Regexp = regexp.MustCompile("url\\((['\"]?)[ \\t\\f]*([\u0009\u0021\u0023-\u0026\u0028\u002a-\u007E]+)(['\"]?)\\)?")
type Proxy struct {
@ -207,49 +76,17 @@ type HTMLMainPageFormParam struct {
URLParamName string
}
var FAVICON_BYTES []byte
var HTML_FORM_EXTENSION *template.Template
var HTML_BODY_EXTENSION *template.Template
var HTML_MAIN_PAGE_FORM *template.Template
var HTML_HEAD_CONTENT_TYPE string = `<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="referrer" content="no-referrer">
`
var YUKARI_HTML_PAGE_START string = `<!doctype html>
<html>
<head>
<title>Yukari's Gap</title>
<meta name="viewport" content="width=device-width, initial-scale=1 , maximum-scale=1.0, user-scalable=1" />
<style>
html { height: 100%; }
body { min-height : 100%; display: flex; flex-direction:column; font-family: sans-serif; text-align: center; color: #BC4BFC; background: #240039; margin: 0;
padding: 0; font-size: 1.1em; }
input { border: 1px solid #888; padding: 0.3em; color: #BC4BFC; background: #202020; font-size: 1.1em; }
input[placeholder] { width:80%; }
a { text-decoration: none; #9529B9; }
h1, h2 { font-weight: 200; margin-bottom: 2rem; }
h1 { font-size: 3em; }
.container { flex:1; min-height: 100%; margin-bottom: 1em; }
.footer { margin: 1em; }
.footer p { font-size: 0.8em; }
</style>
</head>
<body>
<div class="container">
<h1>Yukari's Gap</h1>
`
var YUKARI_HTML_PAGE_END string = `
</div>
<div class="footer">
<p>Yukari rewrites web pages to exclude malicious HTML tags and CSS/HTML attributes. It also replaces external resource references to prevent third-party information leaks.<br />
<a href="https://git.chaotic.ninja/usr/yakumo_izuru/yukari">view on 「混沌とした 忍者」Git</a>
</p>
</div>
</body>
</html>`
var FAVICON_BYTES []byte
//go:embed templates/yukari_content_type.html
var HTML_HEAD_CONTENT_TYPE string
//go:embed templates/yukari_start.html
var YUKARI_HTML_PAGE_START string
//go:embed templates/yukari_stop.html
var YUKARI_HTML_PAGE_END string
func init() {
FaviconBase64 := "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQEAYAAABPYyMiAAAABmJLR0T///////8JWPfcAAAACXBIWXMAAABIAAAASABGyWs+AAAAF0lEQVRIx2NgGAWjYBSMglEwCkbBSAcACBAAAeaR9cIAAAAASUVORK5CYII"
@ -268,7 +105,7 @@ func init() {
<label for="yukaritoggle">hide</label>
<span><a href="/">Yukari's Gap</a></span>
<input type="url" value="{{.BaseURL}}" name="{{.URLParamName}}" {{if .HasYukariKey }}readonly="true"{{end}} />
This is a <a href="https://git.chaotic.ninja/usr/yakumo_izuru/yukari">proxified and sanitized</a> view of the page, visit <a href="{{.BaseURL}}" rel="noreferrer">original site</a>.
This is a proxified and sanitized view of the page, visit <a href="{{.BaseURL}}" rel="noreferrer">original site.
</form>
</div>
<style>
@ -303,9 +140,9 @@ func (p *Proxy) RequestHandler(ctx *fasthttp.RequestCtx) {
return
}
requestHash := popRequestParam(ctx, []byte(cfg.HashParameter))
requestHash := popRequestParam(ctx, []byte(config.Config.HashParameter))
requestURI := popRequestParam(ctx, []byte(cfg.UrlParameter))
requestURI := popRequestParam(ctx, []byte(config.Config.UrlParameter))
if requestURI == nil {
p.serveMainPage(ctx, 200, nil)
@ -315,7 +152,7 @@ func (p *Proxy) RequestHandler(ctx *fasthttp.RequestCtx) {
if p.Key != nil {
if !verifyRequestURI(requestURI, requestHash, p.Key) {
// HTTP status code 403 : Forbidden
error_message := fmt.Sprintf(`invalid "%s" parameter. hint: Hash URL Parameter`, cfg.HashParameter)
error_message := fmt.Sprintf(`invalid "%s" parameter. hint: Hash URL Parameter`, config.Config.HashParameter)
p.serveMainPage(ctx, 403, errors.New(error_message))
return
}
@ -353,7 +190,7 @@ func (p *Proxy) ProcessUri(ctx *fasthttp.RequestCtx, requestURIStr string, redir
}
// Serve an intermediate page for protocols other than HTTP(S)
if (parsedURI.Scheme != "http" && parsedURI.Scheme != "https") || strings.HasSuffix(parsedURI.Host, ".onion") {
if (parsedURI.Scheme != "http" && parsedURI.Scheme != "https") || strings.HasSuffix(parsedURI.Host, ".onion") || strings.HasSuffix(parsedURI.Host, ".i2p") {
p.serveExitYukariPage(ctx, parsedURI)
return
}
@ -362,7 +199,7 @@ func (p *Proxy) ProcessUri(ctx *fasthttp.RequestCtx, requestURIStr string, redir
defer fasthttp.ReleaseRequest(req)
req.SetConnectionClose()
if cfg.Debug {
if config.Config.Debug {
log.Println(string(ctx.Method()), requestURIStr)
}
@ -398,7 +235,7 @@ func (p *Proxy) ProcessUri(ctx *fasthttp.RequestCtx, requestURIStr string, redir
if p.FollowRedirect && ctx.IsGet() {
// GET method: Yukari follows the redirect
if redirectCount < MAX_REDIRECT_COUNT {
if cfg.Debug {
if config.Config.Debug {
log.Println("follow redirect to", string(loc))
}
p.ProcessUri(ctx, string(loc), redirectCount+1)
@ -413,7 +250,7 @@ func (p *Proxy) ProcessUri(ctx *fasthttp.RequestCtx, requestURIStr string, redir
if err == nil {
ctx.SetStatusCode(resp.StatusCode())
ctx.Response.Header.Add("Location", url)
if cfg.Debug {
if config.Config.Debug {
log.Println("redirect to", string(loc))
}
return
@ -503,13 +340,13 @@ func (p *Proxy) ProcessUri(ctx *fasthttp.RequestCtx, requestURIStr string, redir
rc := &RequestConfig{Key: p.Key, BaseURL: parsedURI}
sanitizeHTML(rc, ctx, responseBody)
if !rc.BodyInjected {
p := HTMLBodyExtParam{rc.BaseURL.String(), false, cfg.UrlParameter}
p := HTMLBodyExtParam{rc.BaseURL.String(), false, config.Config.UrlParameter}
if len(rc.Key) > 0 {
p.HasYukariKey = true
}
err := HTML_BODY_EXTENSION.Execute(ctx, p)
if err != nil {
if cfg.Debug {
if config.Config.Debug {
fmt.Println("failed to inject body extension", err)
}
}
@ -595,7 +432,7 @@ func sanitizeCSS(rc *RequestConfig, out io.Writer, css []byte) {
out.Write(css[startIndex:urlStart])
out.Write([]byte(uri))
startIndex = urlEnd
} else if cfg.Debug {
} else if config.Config.Debug {
log.Println("cannot proxify css uri:", string(css[urlStart:urlEnd]))
}
}
@ -714,9 +551,9 @@ func sanitizeHTML(rc *RequestConfig, out io.Writer, htmlDoc []byte) {
if rc.Key != nil {
key = hash(urlStr, rc.Key)
}
err := HTML_FORM_EXTENSION.Execute(out, HTMLFormExtParam{urlStr, key, cfg.UrlParameter, cfg.HashParameter})
err := HTML_FORM_EXTENSION.Execute(out, HTMLFormExtParam{urlStr, key, config.Config.UrlParameter, config.Config.HashParameter})
if err != nil {
if cfg.Debug {
if config.Config.Debug {
fmt.Println("failed to inject body extension", err)
}
}
@ -727,13 +564,13 @@ func sanitizeHTML(rc *RequestConfig, out io.Writer, htmlDoc []byte) {
writeEndTag := true
switch string(tag) {
case "body":
p := HTMLBodyExtParam{rc.BaseURL.String(), false, cfg.UrlParameter}
p := HTMLBodyExtParam{rc.BaseURL.String(), false, config.Config.UrlParameter}
if len(rc.Key) > 0 {
p.HasYukariKey = true
}
err := HTML_BODY_EXTENSION.Execute(out, p)
if err != nil {
if cfg.Debug {
if config.Config.Debug {
fmt.Println("failed to inject body extension", err)
}
}
@ -872,7 +709,7 @@ func sanitizeAttr(rc *RequestConfig, out io.Writer, attrName, attrValue, escaped
case "src", "href", "action":
if uri, err := rc.ProxifyURI(attrValue); err == nil {
fmt.Fprintf(out, " %s=\"%s\"", attrName, uri)
} else if cfg.Debug {
} else if config.Config.Debug {
log.Println("cannot proxify uri:", string(attrValue))
}
case "style":
@ -998,9 +835,9 @@ func (rc *RequestConfig) ProxifyURI(uri []byte) (string, error) {
yukari_uri := u.String()
if rc.Key == nil {
return fmt.Sprintf("./?%s=%s%s", cfg.UrlParameter, url.QueryEscape(yukari_uri), fragment), nil
return fmt.Sprintf("./?%s=%s%s", config.Config.UrlParameter, url.QueryEscape(yukari_uri), fragment), nil
}
return fmt.Sprintf("./?%s=%s&%s=%s%s", cfg.HashParameter, hash(yukari_uri, rc.Key), cfg.UrlParameter, url.QueryEscape(yukari_uri), fragment), nil
return fmt.Sprintf("./?%s=%s&%s=%s%s", config.Config.HashParameter, hash(yukari_uri, rc.Key), config.Config.UrlParameter, url.QueryEscape(yukari_uri), fragment), nil
}
func inArray(b []byte, a [][]byte) bool {
@ -1022,7 +859,7 @@ func verifyRequestURI(uri, hashMsg, key []byte) bool {
h := make([]byte, hex.DecodedLen(len(hashMsg)))
_, err := hex.Decode(h, hashMsg)
if err != nil {
if cfg.Debug {
if config.Config.Debug {
log.Println("hmac error:", err)
}
return false
@ -1050,7 +887,7 @@ func (p *Proxy) serveMainPage(ctx *fasthttp.RequestCtx, statusCode int, err erro
ctx.SetStatusCode(statusCode)
ctx.Write([]byte(YUKARI_HTML_PAGE_START))
if err != nil {
if cfg.Debug {
if config.Config.Debug {
log.Println("error:", err)
}
ctx.Write([]byte("<h2>Error: "))
@ -1058,10 +895,10 @@ func (p *Proxy) serveMainPage(ctx *fasthttp.RequestCtx, statusCode int, err erro
ctx.Write([]byte("</h2>"))
}
if p.Key == nil {
p := HTMLMainPageFormParam{cfg.UrlParameter}
p := HTMLMainPageFormParam{config.Config.UrlParameter}
err := HTML_MAIN_PAGE_FORM.Execute(ctx, p)
if err != nil {
if cfg.Debug {
if config.Config.Debug {
fmt.Println("failed to inject main page form", err)
}
}
@ -1072,49 +909,49 @@ func (p *Proxy) serveMainPage(ctx *fasthttp.RequestCtx, statusCode int, err erro
}
func main() {
listenAddress := flag.String("listen", cfg.ListenAddress, "Listen address")
key := flag.String("key", cfg.Key, "HMAC url validation key (base64 encoded) - leave blank to disable validation")
IPV6 := flag.Bool("ipv6", cfg.IPV6, "Allow IPv6 HTTP requests")
debug := flag.Bool("debug", cfg.Debug, "Debug mode")
requestTimeout := flag.Uint("timeout", cfg.RequestTimeout, "Request timeout")
followRedirect := flag.Bool("followredirect", cfg.FollowRedirect, "Follow HTTP GET redirect")
proxyenv := flag.Bool("proxyenv", false, "Use a HTTP proxy as set in the environment (HTTP_PROXY, HTTPS_PROXY and NO_PROXY). Overrides -proxy, -socks5, -ipv6.")
proxy := flag.String("proxy", "", "Use the specified HTTP proxy (ie: '[user:pass@]hostname:port'). Overrides -socks5, -ipv6.")
socks5 := flag.String("socks5", "", "Use a SOCKS5 proxy (ie: 'hostname:port'). Overrides -ipv6.")
urlParameter := flag.String("urlparam", cfg.UrlParameter, "user-defined requesting string URL parameter name (ie: '/?url=...' or '/?u=...')")
hashParameter := flag.String("hashparam", cfg.HashParameter, "user-defined requesting string HASH parameter name (ie: '/?hash=...' or '/?h=...')")
version := flag.Bool("version", false, "Show version")
var configFile string
var proxy string
var proxyEnv bool
var socks5 string
var version bool
flag.StringVar(&configFile, "f", "", "Configuration file")
flag.BoolVar(&proxyEnv, "proxyenv", false, "Use a HTTP proxy as set in the environment (HTTP_PROXY, HTTPS_PROXY and NO_PROXY). Overrides: -proxy, -socks5, IPv6")
flag.StringVar(&proxy, "proxy", "", "Use the specified HTTP proxy (ie: '[user:pass@]hostname:port'). Overrides: -socks5, IPv6")
flag.StringVar(&socks5, "socks5", "", "Use a SOCKS5 proxy (ie: 'hostname:port'). Overrides: IPv6.")
flag.BoolVar(&version, "version", false, "Show version")
flag.Parse()
cfg.ListenAddress = *listenAddress
cfg.Key = *key
cfg.IPV6 = *IPV6
cfg.Debug = *debug
cfg.RequestTimeout = *requestTimeout
cfg.FollowRedirect = *followRedirect
cfg.UrlParameter = *urlParameter
cfg.HashParameter = *hashParameter
config.Config.ListenAddress = "127.0.0.1:3000"
config.Config.Key = ""
config.Config.IPV6 = true
config.Config.Debug = false
config.Config.RequestTimeout = 5
config.Config.FollowRedirect = false
config.Config.UrlParameter = "yukariurl"
config.Config.HashParameter = "yukarihash"
config.Config.MaxConnsPerHost = 4
if *version {
fmt.Println(VERSION)
if version {
fmt.Println(yukari.FullVersion())
return
}
if *proxyenv && os.Getenv("HTTP_PROXY") == "" && os.Getenv("HTTPS_PROXY") == "" {
if proxyEnv && os.Getenv("HTTP_PROXY") == "" && os.Getenv("HTTPS_PROXY") == "" {
log.Fatal("Error -proxyenv is used but no environment variables named 'HTTP_PROXY' and/or 'HTTPS_PROXY' could be found.")
os.Exit(1)
}
if *proxyenv {
if proxyEnv {
CLIENT.Dial = fasthttpproxy.FasthttpProxyHTTPDialer()
log.Println("Using environment defined proxy(ies).")
} else if *proxy != "" {
CLIENT.Dial = fasthttpproxy.FasthttpHTTPDialer(*proxy)
} else if proxy != "" {
CLIENT.Dial = fasthttpproxy.FasthttpHTTPDialer(proxy)
log.Println("Using custom HTTP proxy.")
} else if *socks5 != "" {
CLIENT.Dial = fasthttpproxy.FasthttpSocksDialer(*socks5)
} else if socks5 != "" {
CLIENT.Dial = fasthttpproxy.FasthttpSocksDialer(socks5)
log.Println("Using Socks5 proxy.")
} else if cfg.IPV6 {
} else if config.Config.IPV6 {
CLIENT.Dial = fasthttp.DialDualStack
log.Println("Using dual stack (IPv4/IPv6) direct connections.")
} else {
@ -1122,21 +959,21 @@ func main() {
log.Println("Using IPv4 only direct connections.")
}
p := &Proxy{RequestTimeout: time.Duration(cfg.RequestTimeout) * time.Second,
FollowRedirect: cfg.FollowRedirect}
p := &Proxy{RequestTimeout: time.Duration(config.Config.RequestTimeout) * time.Second,
FollowRedirect: config.Config.FollowRedirect}
if cfg.Key != "" {
if config.Config.Key != "" {
var err error
p.Key, err = base64.StdEncoding.DecodeString(cfg.Key)
p.Key, err = base64.StdEncoding.DecodeString(config.Config.Key)
if err != nil {
log.Fatal("Error parsing -key", err.Error())
os.Exit(1)
}
}
log.Println("ゆかり様、お願いします…！")
log.Println("Listening on", config.Config.ListenAddress)
log.Println("listening on", cfg.ListenAddress)
if err := fasthttp.ListenAndServe(cfg.ListenAddress, p.RequestHandler); err != nil {
if err := fasthttp.ListenAndServe(config.Config.ListenAddress, p.RequestHandler); err != nil {
log.Fatal("Error in ListenAndServe:", err)
}
}

0
yukari_test.go → cmd/yukari/main_test.go
View File

38
cmd/yukari/safe_attributes.go Normal file
View File

@ -0,0 +1,38 @@
package main
var SAFE_ATTRIBUTES [][]byte = [][]byte{
[]byte("abbr"),
[]byte("accesskey"),
[]byte("align"),
[]byte("alt"),
[]byte("as"),
[]byte("autocomplete"),
[]byte("charset"),
[]byte("checked"),
[]byte("class"),
[]byte("content"),
[]byte("contenteditable"),
[]byte("contextmenu"),
[]byte("dir"),
[]byte("for"),
[]byte("height"),
[]byte("hidden"),
[]byte("hreflang"),
[]byte("id"),
[]byte("lang"),
[]byte("media"),
[]byte("method"),
[]byte("name"),
[]byte("nowrap"),
[]byte("placeholder"),
[]byte("property"),
[]byte("rel"),
[]byte("spellcheck"),
[]byte("tabindex"),
[]byte("target"),
[]byte("title"),
[]byte("translate"),
[]byte("type"),
[]byte("value"),
[]byte("width"),
}

3
cmd/yukari/templates/yukari_content_type.html Normal file
View File

@ -0,0 +1,3 @@
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="referrer" content="no-referrer">

60
cmd/yukari/templates/yukari_start.html Normal file
View File

@ -0,0 +1,60 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1.0, user-scalable=1">
<style>
html {
height: 100%;
}
body {
min-height: 100%;
display: flex;
flex-direction: column;
font-family: sans-serif;
text-align: center;
color: #BC48FC;
background: #240039;
margin: 0;
padding: 0;
font-size: 1.1em;
}
input {
border: 1px solid #888;
padding: 0.3em;
color: #BC48FC;
background: #202020;
font-size: 1.1.em;
}
input[placeholder] {
width: 80%;
}
a {
text-decoration: none;
color: #9529B9;
}
h1, h2 {
font-weight: 200;
margin-bottom: 2rem;
}
h1 {
font-size: 3em;
}
.container {
flex: 1;
min-height: 100%;
margin-bottom: 1em;
}
.footer {
margin: 1em;
}
.footer p {
font-size: 0.8em;
}
</style>
<title>Yukari's Gap</title>
</head>
<body>
<div class="container">
<h1>Yukari's Gap</h1>

10
cmd/yukari/templates/yukari_stop.html Normal file
View File

@ -0,0 +1,10 @@
</div>
<div class="footer">
<p>
Yukari's Gap rewrites web pages to exclude malicious HTML tags and CSS/HTML attributes. <br>
It also replaces external resource references to prevent third-party information leaks. <br>
<a href="https://git.chaotic.ninja/usr/yakumo_izuru/yukari/">View on UnreliableGit</a> &bull; a <a href="https://mirage.h0stname.net">Mirage AIB</a> project
</p>
</div>
</body>
</html>

10
cmd/yukari/unsafe_elements.go Normal file
View File

@ -0,0 +1,10 @@
package main
var UNSAFE_ELEMENTS [][]byte = [][]byte{
[]byte("applet"),
[]byte("canvas"),
[]byte("embed"),
[]byte("math"),
[]byte("script"),
[]byte("svg"),
}

45
config/config.go
View File

@ -1,10 +1,10 @@
package config
import (
"os"
"gopkg.in/ini.v1"
)
type Config struct {
var Config struct {
Debug bool
ListenAddress string
Key string
@ -16,31 +16,20 @@ type Config struct {
HashParameter string
}
var DefaultConfig *Config
func readConfig(file string) error {
cfg, err := ini.Load(file)
if err != nil {
return err
}
Config.Debug, _ = cfg.Section("yukari").Key("debug").Bool()
Config.ListenAddress = cfg.Section("yukari").Key("listen").String()
Config.Key = cfg.Section("yukari").Key("key").String()
Config.IPV6, _ = cfg.Section("yukari").Key("ipv6").Bool()
Config.RequestTimeout, _ = cfg.Section("yukari").Key("timeout").Uint()
Config.FollowRedirect, _ = cfg.Section("yukari").Key("followredirect").Bool()
Config.MaxConnsPerHost, _ = cfg.Section("yukari").Key("max_conns_per_host").Uint()
Config.UrlParameter = cfg.Section("yukari").Key("urlparam").String()
Config.HashParameter = cfg.Section("yukari").Key("hashparam").String()
func init() {
default_listen_addr := os.Getenv("YUKARI_ADDRESS")
if default_listen_addr == "" {
default_listen_addr = "127.0.0.1:3000"
}
default_url_parameter := os.Getenv("YUKARI_URL_PARAM")
if default_url_parameter == "" {
default_url_parameter = "yukariurl"
}
default_hash_parameter := os.Getenv("YUKARI_HASH_PARAM")
if default_hash_parameter == "" {
default_hash_parameter = "yukarihash"
}
default_key := os.Getenv("YUKARI_KEY")
DefaultConfig = &Config{
Debug: os.Getenv("DEBUG") != "false",
ListenAddress: default_listen_addr,
Key: default_key,
IPV6: true,
RequestTimeout: 5,
FollowRedirect: false,
MaxConnsPerHost: 4,
UrlParameter: default_url_parameter,
HashParameter: default_hash_parameter,
}
return nil
}

4
go.mod
View File

@ -1,9 +1,11 @@
module marisa.chaotic.ninja/yukari
go 1.14
go 1.16
require (
github.com/stretchr/testify v1.9.0 // indirect
github.com/valyala/fasthttp v1.34.0
golang.org/x/net v0.7.0
golang.org/x/text v0.7.0
gopkg.in/ini.v1 v1.67.0
)

20
go.sum
View File

@ -1,7 +1,21 @@
github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/klauspost/compress v1.15.0 h1:xqfchp4whNFxn5A4XFyyYtitiWI8Hy5EW59jEwcyL6U=
github.com/klauspost/compress v1.15.0/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.34.0 h1:d3AAQJ2DRcxJYHm7OXNXtXt2as1vMDfxeIcFvhmGGm4=
@ -43,3 +57,9 @@ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGm
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

18
version.go Normal file
View File

@ -0,0 +1,18 @@
package yukari
import (
"fmt"
)
var (
// Version release version
Version = "0.0.1"
// Commit will be overwritten automatically by the build system
Commit = "HEAD"
)
// FullVersion display the full version and build
func FullVersion() string {
return fmt.Sprintf("%s@%s", Version, Commit)
}

117
yukari.1
View File

@ -1,59 +1,64 @@
.TH MORTY "1" "2018" "yukari" "User Commands"
.SH NAME
yukari \- Privacy aware web content sanitizer proxy as a service
.SH SYNOPSIS
.B yukari
.RI [ OPTIONS ]
.br
.SH DESCRIPTION
Yukari rewrites web pages to exclude malicious HTML tags and attributes. It
also replaces external resource references to prevent third party
.Dd $Mdocdate$
.Dt YUKARI 1
.Os
.Sh NAME
.Nm yukari
.Nd Privacy-aware Web Content Sanitizer Proxy As A Service (WCSPAAS)
.Sh SYNOPSIS
.Nm
.Op Fl f Ar string
.Op Fl proxy Ar string
.Op Fl proxyenv Ar bool
.Op Fl socks5 Ar string
.Op Fl version
.Sh DESCRIPTION
Yukari's Gap rewrites web pages to exclude malicious HTML tags and attributes.
It also replaces external resource references in order to prevent third-party
information leaks.
.sp
The main goal of yukari is to provide a result proxy for searx, but it can be
used as a standalone sanitizer service too.
.SH OPTIONS
.HP
\fB\-ipv6\fR
.IP
Allow IPv6 HTTP requests
.HP
\fB\-key\fR string
.IP
HMAC url validation key (hexadecimal encoded) \- leave blank to disable
.HP
\fB\-listen\fR string
.IP
Listen address (default "127.0.0.1:3000")
.HP
\fB\-timeout\fR uint
.IP
Request timeout (default 2)
.HP
\fB\-version\fR
.IP
Show version
.SH BUGS
Bugs or suggestions? Visit the issue tracker at
https://git.chaotic.ninja/yakumo.izuru/yukari/issues.
.SH SEE ALSO
.BR searx (1)
.SH LICENSE
Copyright 2023-present Izuru Yakumo <yakumo.izuru@chaotic.ninja>
.br
Copyright 2016-2018 Adam Tauber <asciimoo@gmail.com>
.br
Copyright 2016 Alexandre Flament <alex@al-f.net>
.sp
This program is free software: you can redistribute it and/or modify it under
the terms of the GNU Affero General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your option) any
later version.
.sp
.Pp
The main goal of Yukari's Gap is to provide a result proxy for SearX, but it
can be used as a standalone sanitizer service, too.
.Sh OPTIONS
.Bl -tag -width Ds
.It Fl f Ar path
Load configuration file from path
.It Fl proxy Ar string
Use the specified HTTP proxy (ie: [user:pass@]hostname:port),
this overrides the
.Fl socks5
option and the IPv6 setting
.It Fl proxyenv Ar bool
Use a HTTP proxy as set in the environment (such as
.Ev HTTP_PROXY ,
.Ev HTTPS_PROXY ,
.Ev NO_PROXY
).
Overrides the
.Fl proxy ,
.Fl socks5 ,
flags and the IPv6 setting
.It Fl socks5 Ar string
Use a SOCKS5 proxy (ie: hostname:port), this
overrides the IPv6 setting
.El
.Sh SEE ALSO
.Xr SearX 1
.Sh AUTHORS
.An Adam Tauber Aq Mt asciimoo@gmail.com
.An Alexandre Flament Aq Mt alex@al-f.net
.Sh MAINTAINERS
.An Izuru Yakumo Aq Mt yakumo.izuru@chaotic.ninja
.Sh BUGS
Bugs or suggestions?
Send an email to
.Aq Mt yukari-dev@chaotic.ninja
.Sh LICENSE
This program is free software: you can redistribute it and/or modify it
under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.Pp
This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU Affero General Public License for more
details.
.sp
You should have received a copy of the GNU Affero General Public License along
with this program. If not, see <http://www.gnu.org/licenses/>.
PARTICULAR PURPOSE.
See the GNU Affero General Public License for more details.