Asked  7 Months ago    Answers:  5   Viewed   43 times

Many of us need to deal with user input, search queries, and situations where the input text can potentially contain profanity or undesirable language. Oftentimes this needs to be filtered out.

Where can one find a good list of swear words in various languages and dialects?

Are there APIs available to sources that contain good lists? Or maybe an API that simply says "yes this is clean" or "no this is dirty" with some parameters?

What are some good methods for catching folks trying to trick the system, like a$$, azz, or a55?

Bonus points if you offer solutions for PHP. :)

Edit: Response to answers that say simply avoid the programmatic issue:

I think there is a place for this kind of filter when, for instance, a user can use public image search to find pictures that get added to a sensitive community pool. If they can search for "penis", then they will likely get many pictures of, yep. If we don't want pictures of that, then preventing the word as a search term is a good gatekeeper, though admittedly not a foolproof method. Getting the list of words in the first place is the real question.

So I'm really referring to a way to figure out of a single token is dirty or not and then simply disallow it. I'd not bother preventing a sentiment like the totally hilarious "long necked giraffe" reference. Nothing you can do there. :)

 Answers

23

Obscenity Filters: Bad Idea, or Incredibly Intercoursing Bad Idea?

Also, one can't forget The Untold History of Toontown's SpeedChat, where even using a "safe-word whitelist" resulted in a 14 year old quickly circumventing it with: "I want to stick my long-necked Giraffe up your fluffy white bunny."

Bottom line: Ultimately, for any system that you implement, there is absolutely no substitute for human review (whether peer or otherwise). Feel free to implement a rudimentary tool to get rid of the drive-by's, but for the determined troll, you absolutely must have a non-algorithm-based approach.

A system that removes anonymity and introduces accountability (something that Stack Overflow does well) is helpful also, particularly in order to help combat John Gabriel's G.I.F.T.

You also asked where you can get profanity lists to get you started -- one open-source project to check out is Dansguardian -- check out the source code for their default profanity lists. There is also an additional third party Phrase List that you can download for the proxy that may be a helpful gleaning point for you.

Edit in response the question edit: Thanks for the clarification on what you're trying to do. In that case, if you're just trying to do a simple word filter, there are two ways you can do it. One is to create a single long regexp with all of the banned phrases that you want to censor, and merely do a regex find/replace with it. A regex like:

$filterRegex = "(boogers|snot|poop|shucks|argh)"

and run it on your input string using preg_match() to wholesale test for a hit,

or preg_replace() to blank them out.

You can also load those functions up with arrays rather than a single long regex, and for long word lists, it may be more manageable. See the preg_replace() for some good examples as to how arrays can be used flexibly.

For additional PHP programming examples, see this page for a somewhat advanced generic class for word filtering that *'s out the center letters from censored words, and this previous Stack Overflow question that also has a PHP example (the main valuable part in there is the SQL-based filtered word approach -- the leet-speak compensator can be dispensed with if you find it unnecessary).

You also added: "Getting the list of words in the first place is the real question." -- in addition to some of the previous Dansgaurdian links, you may find this handy .zip of 458 words to be helpful.

Wednesday, March 31, 2021
 
dkcwd
answered 7 Months ago
73

mysql_real_escape_string won't escape slashes. Even escapeshellarg won't do it. You will have to use str_replace:

$path = str_replace('/', '_', $path);
Saturday, May 29, 2021
 
IvanH
answered 5 Months ago
99

Please don't use this recipe if your situation is not the one described in the question. This recipe is for fixing a bad merge, and replaying your good commits onto a fixed merge.

Although filter-branch will do what you want, it is quite a complex command and I would probably choose to do this with git rebase. It's probably a personal preference. filter-branch can do it in a single, slightly more complex command, whereas the rebase solution is performing the equivalent logical operations one step at a time.

Try the following recipe:

# create and check out a temporary branch at the location of the bad merge
git checkout -b tmpfix <sha1-of-merge>

# remove the incorrectly added file
git rm somefile.orig

# commit the amended merge
git commit --amend

# go back to the master branch
git checkout master

# replant the master branch onto the corrected merge
git rebase tmpfix

# delete the temporary branch
git branch -d tmpfix

(Note that you don't actually need a temporary branch, you can do this with a 'detached HEAD', but you need to take a note of the commit id generated by the git commit --amend step to supply to the git rebase command rather than using the temporary branch name.)

Tuesday, June 1, 2021
 
Sagar
answered 5 Months ago
20

Here's a working example of Lucene.NET using a custom filter you might take a look at:

using System;
using System.Collections;
using Lucene.Net.Analysis;
using Lucene.Net.Documents;
using Lucene.Net.Index;
using Lucene.Net.Search;
using Lucene.Net.Store;

class Program
{
    static void Main(string[] args)
    {
        Directory index = new RAMDirectory();
        Analyzer analyzer = new KeywordAnalyzer();
        IndexWriter writer = new IndexWriter(index, analyzer, true);

        Document doc = new Document();
        doc.Add(new Field("title", "t1", Field.Store.YES, 
            Field.Index.TOKENIZED));
        writer.AddDocument(doc);
        doc = new Document();
        doc.Add(new Field("title", "t2", Field.Store.YES, 
            Field.Index.TOKENIZED));
        writer.AddDocument(doc);

        writer.Close();

        Searcher searcher = new IndexSearcher(index);
        Query query = new MatchAllDocsQuery();
        Filter filter = new LuceneCustomFilter();
        Sort sort = new Sort("title", true);
        Hits hits = searcher.Search(query, filter, sort);
        IEnumerator hitsEnumerator = hits.Iterator();

        while (hitsEnumerator.MoveNext())
        {
            Hit hit = (Hit)hitsEnumerator.Current;
            Console.WriteLine(hit.GetDocument().GetField("title").
                StringValue());
        }
    }
}

public class LuceneCustomFilter : Filter
{
    public override BitArray Bits(IndexReader indexReader)
    {
        BitArray bitarray = new BitArray(indexReader.MaxDoc());

        int[] docs = new int[1];
        int[] freq = new int[1];

        TermDocs termDocs = indexReader.TermDocs(
                new Term(@"title", "t1"));

        int count = termDocs.Read(docs, freq);
        if (count == 1)
        {
            bitarray.Set(docs[0], true);
        }
        return bitarray;
    }
}
Sunday, August 1, 2021
 
Lawrence Taur
answered 3 Months ago
51

I've been using HTMLPurifier for sanitizing the output of a rich text editor, and ended up with:

include_once('htmlpurifier/library/HTMLPurifier.auto.php');

$config = HTMLPurifier_Config::createDefault();
$config->set('Core', 'Encoding', 'UTF-8');
$config->set('HTML', 'Doctype', 'HTML 4.01 Transitional');

if (defined('PURIFIER_CACHE')) {
    $config->set('Cache', 'SerializerPath', PURIFIER_CACHE);
} else {
    # Disable the cache entirely
    $config->set('Cache', 'DefinitionImpl', null);
}

# Help out the Purifier a bit, until it develops this functionality
while (($cleaner = preg_replace('!<(em|strong)>(s*)</1>!', '$2', $input)) != $input) {
    $input = $cleaner;
}

$filter = new HTMLPurifier($config);
$output = $filter->purify($input);

The main points of interest:

  1. Include the autoloader.
  2. Create an instance of HTMLPurifier_Config as $config.
  3. Set configuration settings as needed, with $config->set().
  4. Create an instance of HTMLPurifier, passing $config to it.
  5. Use $filter->purify() on your input.

However, it's entirely overkill for something that doesn't need to allow any HTML in the output.

Monday, August 2, 2021
 
akohout
answered 3 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :
 
Share