Memo's Island: 2025

Wednesday, 12 November 2025

Understanding high-entropy random number generation for research: Leymosun Package

Preamble

The idea of generating randomness via computers goes back to von Neumann's and his team of colleagues work in Princeton and Los Alamos. Even though, he was sceptical about the idea of generating randomness via computer programs, this practice is now almost a de-facto standard, so the name pseudo-random-number-generator (PRNG). Is there a way to generate better PRNGs? This is a continuous research. A practical matter of seeding is often ignored because a good PRNG should be reliable regardless of seeding, however, in repeated calls to a random sequence may degrade this view.

PRNGs bird's eye view : Simulating Randomness

Wigner's Cat (Leymosun Package)

Probably the easiest way to explain RNGs appears to be using discrete recurrence equations, well, actually most RNGs generates numbers based on this, such as $$x_{n+1} = f(x_{n})$$ for example it can be parametrised by $a$ and $b$, $f(x_{n}) = a x_{n} + b$. This basic form allow us remarkably to generate quite useful simulation of randomness.

Concept of a seed

An other concept is a seed, it can be the first number to start recurrence but it can also be some other form of something related to random generation, if $x_{0}$ is fixed or has some complicated formula. We usually put seed like 42 or 4242 or some other integer. There are actually used code with seeds 12345.

Anytime we call a randomisation functions, such as sampling

How eliminating seeding improves things? High-Entropy Randomness

We use new seeds at each new call, this would generate a different sequence and yields us to sample it. This practice would improve the non-predictability as we sample different sequences. But then the question is, where to get the seeds at each call? The answer lies on the "entropy pool" provided by the operating systems, such us Unix's "/dev/random". Essentially they are kind of random devices we tap into. By using hybrid approach of pseudorandom generation via algorithms and seeding from random devices yields to an increased, High-Entropy randomness.

Leymosun: Tested implementation

Leymosun Python package provides this facility, using its random module functionality operates as described above and one doesn't need to provide any seeding, it uses non-deterministic seeding. Using NIST's test, we have shown that high-entropy approach gives higher pass scores in randomness tests compare to baseline seeding, such as 42. To install, just use:

pip install leymosun

Conclusion

Using high-quality randomness for research is quite critical. We have shortly discussed this covering the new approach of using RNGs.

Numerical stability showcase: Ranking with SoftMax or Boltzmann factor

Preamble

Image: Babylonian table for
computation (Wikipedia)

Probably, one of the most important aspects of computational work in quantitative fields, such as physics and data sciences is stability of numerical computations. It implies given inputs, outputs should not wildly deviates to large numbers or it must not distort the results, such as ranking based on scores, one of the most used computation in data science tasks, such as in classification of clustering. In this short post, we provide a stunning example of using SoftMax that creates wrong results if it applied naively.

SoftMax: Normalisation with Boltzmann factor

SoftMax is actually something more physics concept than a data science usage. The most common usage in data science is used for ranking. Given a vector $x_{i}$, then softmax can be computed with the following expression read $$exp(x_{i})/\sum_{i} exp(x_{I}).$$ This originates from statistical physics, i.e., Boltzmann factor.

Source of Numerical Instability

Using exponential function in the denominator in a sum creates a numerical instability, if one of the number deviates from other numbers significantly in the vector. This makes all other entries zero for the output of softmax.

Example Instability: Ranking with SoftMax

Let's say we have the following scores

scores = [1.4, 1.5, 1.6, 170]

for teams A, B, C, D for some metric, we want to turn this into probabilistic interpretation with SoftMax, this will read, [0., 0., 0., 1.] we see that D comes on top but A,B,C are tied.

LogSoftMax

We can rectify this instability by using LogSoftMax. reads $$\log exp(x_{I})-log(\sum_{i} exp(x_{i})),$$reads [-168.6000, -168.5000, -168.4000, 0.0000], so that we can induce consecutive ranking without ties, as follows D, A, B, C.

Conclusion

There is a similar practice in statistics for likelihood computations, as Gaussians brings exponential repeatedly. Using Log of the given operations will stabilise the numerical instabilities caused by repeated exponentiation. This shows the importance of numerical pitfalls in data sciences.

Cite as follows

@misc{suezen25softmax,

title = {Numerical stability showcase: Ranking with SoftMax or Boltzmann factor},

howpublished = {\url{https://memosisland.blogspot.com/2025/04/softmax-numerical-stability.html}},

author = {Mehmet Süzen},

year = {2025}

}

Appendix: Python method

A python method using PyTorch computing softmax example from the main text.

import torch 

List = list
Tensor = torch.tensor

def get_softmax(scores:List, log :bool = False) -> Tensor:
  """ 
  
  Compute softmax of a list

  Defaults to LogSoftMax 
  
  """ 
  scores = torch.tensor(scores)
  if log:
    scores = torch.log_softmax(scores, dim=0)
  else:
    scores = torch.softmax(scores, dim=0)
  return  scores

Memo's Island