Asked  9 Months ago    Answers:  5   Viewed   105 times

I'm selecting some data from database and encoding them as json, but I've got a problem with czech signs like

á,í,?,?,ž...

My file is in utf-8 encoding, my database is also in utf-8 encoding, I've set header to utf-8 encoding as well. What else should I do please?

My code:

header('Content-Type: text/html; charset=utf-8');
while($tmprow = mysqli_fetch_array($result)) {
        $row['user'] = mb_convert_encoding($tmprow['user'], "UTF-8", "auto");
        $row['package'] = mb_convert_encoding($tmprow['package'], "UTF-8", "auto");
        $row['url'] = mb_convert_encoding($tmprow['url'], "UTF-8", "auto");
        $row['rating'] = mb_convert_encoding($tmprow['rating'], "UTF-8", "auto");

        array_push($response, $row);
    }

    $json = json_encode($response, JSON_UNESCAPED_UNICODE);

    if(!$json) {
        echo "error";
    }

and part of the printed json: "package":"zv???tkanalouce"

EDIT: Without mb_convert_encoding() function the printed string is empty and "error" is printed.

 Answers

17

With the code you've got in your example, the output is:

json_encode($response, JSON_UNESCAPED_UNICODE);
"package":"zv???tkanalouce"

You see the question marks in there because they have been introduced by mb_convert_encoding. This happens when you use encoding detection ("auto" as third parameter) and that encoding detection is not able to handle a character in the input, replacing it with a question mark. Exemplary line of code:

$row['url'] = mb_convert_encoding($tmprow['url'], "UTF-8", "auto");

This also means that the data coming out of your database is not UTF-8 encoded because mb_convert_encoding($buffer, 'UTF-8', 'auto'); does not introduce question marks if $buffer is UTF-8 encoded.

Therefore you need to find out which charset is used in your database connection because the database driver will convert strings into the encoding of the connection.

Most easy is that you just tell per that database link that you're asking for UTF-8 strings and then just use them:

$mysqli = new mysqli("localhost", "my_user", "my_password", "test");

/* check connection */
if (mysqli_connect_errno()) {
    printf("Connect failed: %sn", mysqli_connect_error());
    exit();
}

/* change character set to utf8 */
if (!$mysqli->set_charset("utf8")) {
    printf("Error loading character set utf8: %sn", $mysqli->error);
} else {
    printf("Current character set: %sn", $mysqli->character_set_name());
}

The previous code example just shows how to set the default client character set to UTF-8 with mysqli. It has been taken from the manual, see as well the material we have on site about that, e.g. utf 8 - PHP and MySQLi UTF8.

You can then greatly improve your code:

$response = $result->fetch_all(MYSQLI_ASSOC);

$json = json_encode($response, JSON_UNESCAPED_UNICODE);

if (FALSE === $json) {
    throw new LogicException(
        sprintf('Not json: %d - %s', json_last_error(), json_last_error_msg())
    );
}

header('Content-Type: application/json'); 
echo $json;
Wednesday, March 31, 2021
 
mistero
answered 9 Months ago
81

You have probably come to mix encoding types. For example. A page that is sent as iso-8859-1, but get UTF-8 text encoding from MySQL or XML would typically fail.

To solve this problem you must keep control on input ecodings type in relation to the type of encoding you have chosen to use internal.

If you send it as an iso-8859-1, your input from the user is also iso-8859-1.

header("Content-type:text/html; charset: iso-8859-1");

And if mysql sends latin1 you do not have to do anything.

But if your input is not iso-8859-1 you must converted it, before it's sending to the user or to adapt it to Mysql before it's store.

mb_convert_encoding($text, mb_internal_encoding(), 'UTF-8'); // If it's UTF-8 to internal encoding

Short it means that you must always have input converted to fit internal encoding and convereter output to match the external encoding.


This is the internal encoding I have chosen to use.

mb_internal_encoding('iso-8859-1'); // Internal encoding

This is a code i use.

mb_language('uni'); // Mail encoding
mb_internal_encoding('iso-8859-1'); // Internal encoding
mb_http_output('pass'); // Skip

function convert_encoding($text, $from_code='', $to_code='')
{
    if (empty($from_code))
    {
        $from_code = mb_detect_encoding($text, 'auto');
        if ($from_code == 'ASCII')
        {
            $from_code = 'iso-8859-1';
        }
    }

    if (empty($to_code))
    {
        return mb_convert_encoding($text, mb_internal_encoding(), $from_code);
    }
    return mb_convert_encoding($text, $to_code, $from_code);
}

function encoding_html($text, $code='')
{
    if (empty($code))
    {
        return htmlentities($text, ENT_NOQUOTES, mb_internal_encoding());
    }

    return mb_convert_encoding(htmlentities($text, ENT_NOQUOTES, $code), mb_internal_encoding(), $code);
}
function decoding_html($text, $code='')
{
    if (empty($code))
    {
        return html_entity_decode($text, ENT_NOQUOTES, mb_internal_encoding());
    }

    return mb_convert_encoding(html_entity_decode($text, ENT_NOQUOTES, $code), mb_internal_encoding(), $code);
}
Wednesday, March 31, 2021
 
capsid
answered 9 Months ago
78

Well this question is 9 months old so i'm not sure if OP is still in the need of an answer but due the many views and the tasty bounty I would like to also add my mustard (German saying..).

In this post I will try to make a simple explained example on how to start building a notification system.

Edit: Well ok this turned out way, way, way longer than I expected it to be. I got really tired in the end, i'm sorry.

WTLDR;

Question 1: have a flag on every notification.

Question 2: Still store every notification as a single record inside your database and group them when they are requested.


Structure

I assume that the notifications will look something like:

+---------------------------------------------+
| ? James has uploaded new Homework: Math 1+1 |
+---------------------------------------------+
| ? Jane and John liked your comment: Im s... | 
+---------------------------------------------+
| ? The School is closed on independence day. |
+---------------------------------------------+

Behind the curtains this could look something like this:

+--------+-----------+--------+-----------------+-------------------------------------------+
| unread | recipient | sender | type            | reference                                 |
+--------+-----------+--------+-----------------+-------------------------------------------+
| true   | me        | James  | homework.create | Math 1 + 1                                |
+--------+-----------+--------+-----------------+-------------------------------------------+
| true   | me        | Jane   | comment.like    | Im sick of school                         |
+--------+-----------+--------+-----------------+-------------------------------------------+
| true   | me        | John   | comment.like    | Im sick of school                         |
+--------+-----------+--------+-----------------+-------------------------------------------+
| false  | me        | system | message         | The School is closed on independence day. |
+--------+-----------+--------+-----------------+-------------------------------------------+

Note: I don't recommend to group the notifications inside the database, do that on runtime this keeps things a lot more flexible.

  • Unread
    Every notification should have a flag to indicate if the recipient has already opened the notification.
  • Recipient
    Defines who receives the notification.
  • Sender
    Defines who triggered the notification.
  • Type
    Instead of having every Message in plain text inside your database create types. This way you can create special handlers for different notification types inside your backend. Will reduce the amount of data stored inside your database and gives your even more flexibility, enabled easy translating of notification, changes of past messages etc..
  • Reference
    Most notifications will have a Reference to a record on your database or your application.

Every system I have been working on had a simple 1 to 1 reference relationship on a notification, you might have an 1 to n keep in mind that I will continue my example with 1:1. This also means that I don't need a field defining what type of object is referenced because this is defined by the notification type.

SQL Table

Now when defining a real table structure for SQL we come to a few decisions in terms of the database design. I will go with simplest solution which will look something like this:

+--------------+--------+---------------------------------------------------------+
| column       | type   | description                                             |
+--------------+--------+---------------------------------------------------------+
| id           | int    | Primary key                                             |
+--------------+--------+---------------------------------------------------------+
| recipient_id | int    | The receivers user id.                                  |
+--------------+--------+---------------------------------------------------------+
| sender_id    | int    | The sender's user id.                                   |
+--------------+--------+---------------------------------------------------------+
| unread       | bool   | Flag if the recipient has already read the notification |
+--------------+--------+---------------------------------------------------------+
| type         | string | The notification type.                                  |
+--------------+--------+---------------------------------------------------------+
| parameters   | array  | Additional data to render different notification types. |
+--------------+--------+---------------------------------------------------------+
| reference_id | int    | The primary key of the referencing object.              |
+--------------+--------+---------------------------------------------------------+
| created_at   | int    | Timestamp of the notification creation date.            |
+--------------+--------+---------------------------------------------------------+

Or for the lazy folks the SQL create table command for this example:

CREATE TABLE `notifications` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `recipient_id` int(11) NOT NULL,
  `sender_id` int(11) NOT NULL,
  `unread` tinyint(1) NOT NULL DEFAULT '1',
  `type` varchar(255) NOT NULL DEFAULT '',
  `parameters` text NOT NULL,
  `reference_id` int(11) NOT NULL,
  `created_at` int(11) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

PHP Service

This implementation depends completely on the needs of your application, Note: This is an example not the golden standard on how to build an notification system in PHP.

Notification model

This is an example base model of the notification itself, nothing fancy just the needed properties and the abstract methods messageForNotification and messageForNotifications we expected being implemented in the different notification types.

abstract class Notification
{
    protected $recipient;
    protected $sender;
    protected $unread;
    protected $type;
    protected $parameters;
    protected $referenceId;
    protected $createdAt;

    /**
     * Message generators that have to be defined in subclasses
     */
    public function messageForNotification(Notification $notification) : string;
    public function messageForNotifications(array $notifications) : string;

    /**
     * Generate message of the current notification.
     */ 
    public function message() : string
    {
        return $this->messageForNotification($this);
    }
}

You will have to add a constructor, getters, setters and that kind of stuff by yourself in your own style, i'm not going to provide a ready to use Notification system.

Notification Types

Now you can create a new Notification subclass for every type. This following example would handle the like action of a comment:

  • Ray has liked your comment. (1 notification)
  • John and Jane liked your comment. (2 notifications)
  • Jane, Johnny, James and Jenny liked your comment. (4 notifications)
  • Jonny, James and 12 others liked your comment. (14 notifications)

Example implementation:

namespace NotificationComment;

class CommentLikedNotification extends Notification
{
    /**
     * Generate a message for a single notification
     * 
     * @param Notification              $notification
     * @return string 
     */
    public function messageForNotification(Notification $notification) : string 
    {
        return $this->sender->getName() . 'has liked your comment: ' . substr($this->reference->text, 0, 10) . '...'; 
    }

    /**
     * Generate a message for a multiple notifications
     * 
     * @param array              $notifications
     * @return string 
     */
    public function messageForNotifications(array $notifications, int $realCount = 0) : string 
    {
        if ($realCount === 0) {
            $realCount = count($notifications);
        }

        // when there are two 
        if ($realCount === 2) {
            $names = $this->messageForTwoNotifications($notifications);
        }
        // less than five
        elseif ($realCount < 5) {
            $names = $this->messageForManyNotifications($notifications);
        }
        // to many
        else {
            $names = $this->messageForManyManyNotifications($notifications, $realCount);
        }

        return $names . ' liked your comment: ' . substr($this->reference->text, 0, 10) . '...'; 
    }

    /**
     * Generate a message for two notifications
     *
     *      John and Jane has liked your comment.
     * 
     * @param array              $notifications
     * @return string 
     */
    protected function messageForTwoNotifications(array $notifications) : string 
    {
        list($first, $second) = $notifications;
        return $first->getName() . ' and ' . $second->getName(); // John and Jane
    }

    /**
     * Generate a message many notifications
     *
     *      Jane, Johnny, James and Jenny has liked your comment.
     * 
     * @param array              $notifications
     * @return string 
     */
    protected function messageForManyNotifications(array $notifications) : string 
    {
        $last = array_pop($notifications);

        foreach($notifications as $notification) {
            $names .= $notification->getName() . ', ';
        }

        return substr($names, 0, -2) . ' and ' . $last->getName(); // Jane, Johnny, James and Jenny
    }

    /**
     * Generate a message for many many notifications
     *
     *      Jonny, James and 12 other have liked your comment.
     * 
     * @param array              $notifications
     * @return string 
     */
    protected function messageForManyManyNotifications(array $notifications, int $realCount) : string 
    {
        list($first, $second) = array_slice($notifications, 0, 2);

        return $first->getName() . ', ' . $second->getName() . ' and ' .  $realCount . ' others'; // Jonny, James and 12 other
    }
}

Notification manager

To work with your notifications inside your application create something like a notification manager:

class NotificationManager
{
    protected $notificationAdapter;

    public function add(Notification $notification);

    public function markRead(array $notifications);

    public function get(User $user, $limit = 20, $offset = 0) : array;
}

The notificationAdapter property should contain the logic in direct communication with your data backend in the case of this example mysql.

Creating notifications

Using mysql triggers is not wrong, because there is no wrong solution. What works, works.. But I strongly recommend to not let the database handle application logic.

So inside the notification manager you might want to do something like this:

public function add(Notification $notification)
{
    // only save the notification if no possible duplicate is found.
    if (!$this->notificationAdapter->isDoublicate($notification))
    {
        $this->notificationAdapter->add([
            'recipient_id' => $notification->recipient->getId(),
            'sender_id' => $notification->sender->getId()
            'unread' => 1,
            'type' => $notification->type,
            'parameters' => $notification->parameters,
            'reference_id' => $notification->reference->getId(),
            'created_at' => time(),
        ]);
    }
}

Behind the add method of the notificationAdapter can be a raw mysql insert command. Using this adapter abstraction enables you to switch easily from mysql to a document based database like mongodb which would make sense for a Notification system.

The isDoublicate method on the notificationAdapter should simply check if there is already a notification with the same recipient, sender, type and reference.


I cannot point out enough that this is only a example. (Also I really have to shorten the next steps this post is getting ridiculously long -.-)


So assuming you have some kind of controller with an action when a teacher uploads homework:

function uploadHomeworkAction(Request $request)
{
    // handle the homework and have it stored in the var $homework.

    // how you handle your services is up to you...
    $notificationManager = new NotificationManager;

    foreach($homework->teacher->students as $student)
    {
        $notification = new NotificationHomeworkHomeworkUploadedNotification;
        $notification->sender = $homework->teacher;
        $notification->recipient = $student;
        $notification->reference = $homework;

        // send the notification
        $notificationManager->add($notification);
    }
}

Will create a notification for every teacher's student when he uploads a new homework.

Reading the notifications

Now comes the hard part. The problem with grouping on the PHP side is that you will have to load all notifications of the current user to group them correctly. This would be bad, well if you have only a few users it would probably still be no problem, but that doesn't make it good.

The easy solution is to simply limit the number of notifications requested and only grouping these. This will work fine when there are not many similar notifications (like 3-4 per 20). But lets say the post of a user / student gets about a hundred likes and you only select the last 20 notifications. The user will then only see that 20 people liked his post also that would be his only notification.

A "correct" solution would be grouping the notifications already on the database and selecting only some samples per notification group. Than you would just have to inject the real count into your notification messages.

You probably didn't read the text below so let me continue with a snippet:

select *, count(*) as count from notifications
where recipient_id = 1
group by `type`, `reference_id`
order by created_at desc, unread desc
limit 20

Now you know what notifications should be around for the given user and how many notifications the group contains.

And now the shitty part. I still could not find out a better way to select a limited number of notifications for each group without doing a query for every group. All suggestions here are very welcome.

So I do something like:

$notifcationGroups = [];

foreach($results as $notification)
{
    $notifcationGroup = ['count' => $notification['count']];

    // when the group only contains one item we don't 
    // have to select it's children
    if ($notification['count'] == 1)
    {
        $notifcationGroup['items'] = [$notification];
    }
    else
    {
        // example with query builder
        $notifcationGroup['items'] = $this->select('notifications')
            ->where('recipient_id', $recipient_id)
            ->andWehere('type', $notification['type'])
            ->andWhere('reference_id', $notification['reference_id'])
            ->limit(5);
    }

    $notifcationGroups[] = $notifcationGroup;
}

I will now continue assuming that the notificationAdapters get method implements this grouping and returns an array like this:

[
    {
        count: 12,
        items: [Note1, Note2, Note3, Note4, Note5] 
    },
    {
        count: 1,
        items: [Note1] 
    },
    {
        count: 3,
        items: [Note1, Note2, Note3] 
    }
]

Because we always have at least one notification in our group and our ordering prefers Unread and New notifications we can just use the first notification as a sample for rendering.

So to be able to work with these grouped notifications we need a new object:

class NotificationGroup
{
    protected $notifications;

    protected $realCount;

    public function __construct(array $notifications, int $count)
    {
        $this->notifications = $notifications;
        $this->realCount = $count;
    }

    public function message()
    {
        return $this->notifications[0]->messageForNotifications($this->notifications, $this->realCount);
    }

    // forward all other calls to the first notification
    public function __call($method, $arguments)
    {
        return call_user_func_array([$this->notifications[0], $method], $arguments);
    }
}

And finally we can actually put most of the stuff together. This is how the get function on the NotificationManager might look like:

public function get(User $user, $limit = 20, $offset = 0) : array
{
    $groups = [];

    foreach($this->notificationAdapter->get($user->getId(), $limit, $offset) as $group)
    {
        $groups[] = new NotificationGroup($group['notifications'], $group['count']);
    }

    return $gorups;
}

And really finally inside a possible controller action:

public function viewNotificationsAction(Request $request)
{
    $notificationManager = new NotificationManager;

    foreach($notifications = $notificationManager->get($this->getUser()) as $group)
    {
        echo $group->unread . ' | ' . $group->message() . ' - ' . $group->createdAt() . "n"; 
    }

    // mark them as read 
    $notificationManager->markRead($notifications);
}
Wednesday, March 31, 2021
 
fardjad
answered 9 Months ago
46

What you're looking for is the Unicode code point, i.e. the numeric identifier by which the character is known in the Unicode character table. The "cheapest" way to do this is through the UCS-2 character encoding, which maps 1:1 from bytes unto the Unicode code points:

echo bin2hex(iconv('UTF-8', 'UCS-2', '?'));
// 3042

Caveats: the returned code is always 4 hexadecimal digits long (which you may or may not like) and UCS-2 does not support characters higher than the BMP, i.e. higher than code point FFFF.

Wednesday, March 31, 2021
 
RemiX
answered 9 Months ago
16

Apparently in my DbConnect class my PHPDoc comments stated that I was returning 'database' when in fact I was returning a 'mysqli' datatype. This was what was causing the error. The simply fix to this problem was to change be PHPDoc annotation to 'mysqli' and the code completions started to work again.

Saturday, May 29, 2021
 
viper
answered 7 Months ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :
 
Share